High-speed vision sensor having a parallel processing system

ABSTRACT

A high-speed vision sensor has: an analog-to-digital converter array  13  including analog-to-digital converters  210  corresponding to respective lines of photodetectors  120  of a photodetector array  11 , and a parallel processing system  14  including processing elements  400  and shift registers  410 . The processing elements  400  are provided in one to one correspondence with the photodetectors  120 . The shift registers  410  are provided in one to one correspondence with the photodetectors  120 . Because the processing elements  400  carry out the image processing between neighboring pixels by parallel processing at high speed, independently from the operation in the shift registers  410 , the processing and shifting can be performed efficiently.

TECHNICAL FIELD

The present invention relates to a high-speed vision sensor having animage processing function.

BACKGROUND ART

High-speed image processing is required to operate high-speed robots ina factory automation (FA) system or the like. For example, when a robotis configured with a feedback loop between its vision sensor and itsactuator, the actuator can be controlled in units of milliseconds.Accordingly, the image processing speed matching this speed is required.However, because the image processing speed in modern vision systems islimited by the video frame rate, the robot can only operate at lowspeeds matching the image processing speed. It is impossible to takefull advantage of the robot's performing capability.

There are some high-speed CCD cameras that can take images at intervalsof about one (1) millisecond. In these devices, the images aretemporarily stored in memory and later read and processed. Therefore,the devices can be applied to such applications as image analysis.However, the devices have almost no real-time capability, and are notsuitable for controlling robots or similar real-time applications.

In order to overcome these problems, institutes such as theMassachusetts Institute of Technology, the California Institute ofTechnology, and Mitsubishi Electric Corporation have been researching avision chip that combines the image reading portion and the processingportion into one unit (“An Object Position and Orientation IC withEmbedded Imager,” David L. Standley, Solid State Circuits, Vol. 26, No.12, Dec. 1991, pp. 1853-1859, IEEE); “Computing Motion Using Analog andBinary Resistive Networks,” James Hutchinson, et al., Computer, Vol. 21,March 1988, pp. 52-64, IEEE); and “Artificial Retinas—fast versatileimage processors,” Kazuo Kyuma, et al., Nature, Vol. 372, Nov. 10,1994). However, these chips employ a fixed analog circuit that is easyto integrate. Accordingly, these circuits have such shortcomings asrequiring subsequent-processing of output signals and a lack ofuniversality. Hence, the type of image processing they can perform islimited to special applications.

Japanese Unexamined Patent Application Publication HEI-10-145680 hasproposed a vision chip that is capable of performing universal imageprocessing. This vision chip is provided with a processing element foreach photodetector. An analog-to-digital converter is provided for eachphotodetector row. Therefore, the vision chip can reduce the processingtime through parallel processing. The vision chip can also reduce thenumber of transmission lines between the photodetectors and theprocessing elements, achieving an optimal integration level for both.

However, since this vision chip is configured to use the processingelements themselves when transferring data thereto, processing cannot beperformed during image transfers. It is noted that it is necessary toperform image processing over a plurality of images in order to detectthe shape of objects or to detect movement. Since conventional methodstemporarily read a plurality of images into memory before processing,such processing cannot be performed in real-time.

DISCLOSURE OF THE INVENTION

In view of the foregoing, it is an object of the present invention toprovide a multi-pixel high-speed vision sensor which has a simplecircuit construction and which is capable of performing efficienthigh-speed calculations even over a plurality of images.

In order to attain the object, the high-speed vision sensor of thepresent invention comprises: at least one photodetector array, eachhaving a plurality of photodetectors which are arrangedtwo-dimensionally in a plurality of rows and in a plurality of columns;an analog-to-digital converter array having a plurality ofanalog-to-digital converters which are arranged one-dimensionally suchthat each analog-to-digital converter corresponds to one row in the atleast one photodetector array, each analog-to-digital converterconverting, into digital signals, analog signals which are successivelyoutputted from the photodetectors in the corresponding row; a parallelprocessing system, including a parallel processing element array and ashift register array, the parallel processing clement array having aplurality of processing elements which are arranged two-dimensionally ina plurality of rows and in a plurality of columns and in one-to-onecorrespondence with the plurality of photodetectors in the at least onephotodetector array, each processing element performing a predeterminedcalculation on digital signals transferred from the analog-to-digitalconverter array, the shift register array having a plurality of shiftregisters which are disposed in one-to-one correspondence with theplurality of analog-to-digital converters and in one-to-onecorrespondence with the plurality of rows of processing elements, eachshift register successively transferring digital signals, which arereceived from the corresponding analog-to-digital converter and whichare equivalent to signals outputted from the photodetectors in acorresponding photodetector row, to predetermined processing elements inthe corresponding row; and a control circuit controlling thephotodetector array and the analog-to-digital converter array to outputdigital signals for a single frame and controlling the shift registerarray to transfer the digital signals of the single frame to theparallel processing element array, and thereafter controlling thephotodetector array and the analog-to-digital converter array to outputdigital signals for the next frame and controlling the shift registerarray to transfer the digital signals of the next frame to the parallelprocessing element array, while simultaneously controlling the parallelprocessing element array to perform the predetermined calculation ontothe single frame.

According to the present invention, the plurality of processing elementsare provided in one to one correspondence with the plurality ofphotodetectors. It is therefore possible to perform high-speed imageprocessing through parallel processing. Additionally, the shiftregisters are used as being dedicated to transferring data forprocessing. Accordingly, image processing can be performed during thetransfer process. Image processing can be performed efficiently byreducing the wait time for both of the transfer process and thecalculating process, thereby reducing the overall processing time. It istherefore possible to achieve a pipeline operation. It is possible toperform high-speed image processing, and particularly real-timeprocessing. An analog-to-digital converter is provided for each row.Accordingly, the total number of transmission paths can be reduced.

When the high-speed vision sensor has a plurality of photodetectorarrays, the parallel processing system may preferably include, incorrespondence with each processing element row, a plurality of lines ofshift registers, the number of the plurality of lines being equal to thenumber of the plurality of photodetector arrays.

With this construction, the plurality of lines of shift registers areprovided for transferring, to each row of processing elements, data fromthe corresponding row of photodetectors in the plurality ofphotodetector arrays. Accordingly, even when the high-speed visionsensor has plural photodetector arrays, image signals from the pluralphotodetector arrays can be transferred independently from one another,thereby requiring no extra time for the transfer process than when thehigh-speed vision sensor has only one photodetector array. Therefore,the image processes, including the calculating processes, can beperformed at a high rate of speed.

The plurality of photodetector arrays may be disposed at positionsdifferent from one another. The control circuit may include: a parallelprocessing control portion controlling the respective lines of shiftregisters to transfer images which are taken at different positions andoutputted from the corresponding photodetector arrays, and controllingthe parallel processing system to perform calculations, based on imagesignals obtained by the plurality of photodetector arrays, to determinethe amount of positional shift, of an object, between its images takenby the plurality of photodetector arrays; and a calculating portioncalculating three-dimensional positional information of the object basedon the determined amount of positional shift and information on theposition of each photodetector array and on the direction in which eachphotodetector array takes images.

By arranging the plurality of photodetector arrays in differentpositions and by controlling the photodetector arrays to take images ofthe same object, images obtained by the respective photodetector arrayswill correspond to images of the object projected in predetermineddirections. It is possible to determine the three-dimensional positionof the object by using trigonometry based on the positions of the sameportion of the object in the respective images. The above-describedprocess requires at least two photodetector arrays. However, three ormore photodetector arrays can be used in order to improve accuracy andto reduce blind spots.

In order to determine the three-dimensional position of the object, theplurality of photodetector arrays may preferably be arranged such thatdata transfer directions, along which the rows of the photodetectors inthe respective photodetector arrays extend to transfer data from thephotodetectors, are lined up in the same direction and such thatcorresponding photodetectors in the plural photodetector arrays arepositioned to be shifted from one another in the data transferdirection. The parallel processing control portion may control theparallel processing system to calculate, based on the image signalsobtained by the plural photodetector arrays, the amount of positionalshift along the data transfer direction among the images taken by theplural photodetector arrays.

In this way, the direction of data transfer is the same for all thephotodetector arrays, and corresponding photodetectors in thephotodetector arrays are disposed in positions which are shifted fromone another in the data transfer direction. Accordingly, a matchingpoint of the images detected by the photodetector arrays will appear indifferent positions in the images taken by the plural photodetectorarrays. This shift in position will occur in the same direction as thatin which the photodetector arrays are shifted, that is, the direction ofdata transfer. Because the direction in which the shift in positionoccurs is the same as the data transfer direction, it becomes easy toshift one image in relation to another image by controlling the transferamount of data transfer. It therefore becomes easy to search the samepart of images and, therefore, to detect three-dimensional positionalinformation.

Preferably, the high-speed vision sensor may further comprise a pixelshift amount storage device storing the amount of positional shift,along the data transfer direction, which is calculated by the parallelprocessing system with respect to the plurality of images taken by theplurality of photodetector arrays. The parallel processing controlportion may control, based on the stored positional shift amount, thetransfer position of the processing elements, to which each line ofshift registers transfers an image signal.

With this construction, it is possible to perform calculations using, asa reference, the positional shift amount calculated for the previousframe. Hence, calculations of the positional shift amount andrecognition of the matching point can be performed more quickly.

Alternatively, the high-speed vision sensor may further comprise: atiming controller that controls the plurality of photodetector arrays totake images at timings independent from one another; and a beam splitterthat enables the plurality of photodetector arrays to pick up imagesfrom the same direction. The control circuit may include a parallelprocessing control portion controlling the plural lines of shiftregisters to transfer images taken by the respective photodetectorarrays at the different times, and controlling the processing elementsto perform calculation onto the images taken at the different times.

The timings when images are taken by the photodetector arrays arecontrolled independently from one another. Thus, the times when imagesare obtained are controlled. Additionally, the photodetector arrays aremade to take images of the object from the same direction. Accordingly,the plural photodetector arrays can take images at different times atshort intervals. It is possible to obtain a plurality of frame images atdifferent times by transferring those images using the different linesof shift registers. By performing calculations onto those images, it ispossible to easily attain the recognition of a high-speed moving objector of movement.

The high-speed vision sensor may further comprise a filtering/beamsplitter mechanism enabling the plurality of photodetector arrays topick up, from the same direction, color-separated images, of an object,which have colors different from one another. The control circuit mayinclude a parallel processing control portion controlling the plurallines of shift registers to transfer the color-separated imagesoutputted from the respective photodetector arrays and controlling theprocessing elements to perform calculations onto the color-separatedimages.

With this construction, color-separated image data is transferred fromeach photodetector array to the processing elements. Hence, theprocessing elements can easily perform image processes such as coloradjustment or color correction.

The high-speed vision sensor may further comprise data supply mechanismsupplying predetermined data required for image processing calculation.Each shift register may successively transfer output signals, outputtedfrom the corresponding analog-to-digital converter, and thepredetermined data, supplied from the data supply mechanism, to thepredetermined processing elements in the corresponding processingelement row. The control circuit may control the photodetector array,the analog-to-digital converter array, the parallel processing system,and the data supply mechanism.

With this construction, not only image data from the photodetectors, butalso various types of data, necessary for processing, is transferred tothe processing elements using the dedicated shift registers.Accordingly, it is possible to perform these data transfer operation inparallel with the image processing operation.

The data supply mechanism may preferably include a time-sharing mixingunit combining, according to a time-sharing manner, output signalsoutputted from the analog-to-digital converters and the predetermineddata. By combining, in a time-sharing manner, image data and thepredetermined data necessary for processing, both types of data can betransferred reliably and efficiently on the same line of shiftregisters.

Alternatively, the parallel processing system may further include aplurality of data-transfer shift registers which are arranged inone-to-one correspondence with the plurality of rows of processingelements, each data-transfer shift register supplying the predetermineddata to the respective processing elements in the corresponding row.

With this construction, the high-speed vision sensor is provided withtwo separated groups of shift registers: one group being fortransferring input image data; and the other being for transferring thepredetermined data required for image processing. Accordingly, thehigh-speed vision sensor can achieve a pipeline process by performing inparallel: the calculation operation with the processing elements, thetransferring operation of input image data, and the transferringoperation of other data required for image processing, thereby enablingthe high-speed image processing and, particularly, the real time imageprocessing.

When the high-speed vision sensor has a plurality of photodetectorarrays, the parallel processing system may preferably include a singleline of shift register for each processing element row, the single lineof shift register being used in a time-sharing manner to transfer theoutput signals from the plurality of photodetector arrays.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a high-speed vision sensor accordingto a first embodiment of the present invention;

FIG. 2 is a schematic view showing the construction of the high-speedvision sensor of the first embodiment;

FIG. 3(A) is a structural block diagram showing a control circuitemployed in the high-speed vision sensor of the first embodiment;

FIG. 3(B) is a functional block diagram showing the control circuitemployed in the high-speed vision sensor of the first embodiment;

FIG. 4 is a circuit diagram showing a photodetector array and ananalog-to-digital converter array of the high-speed vision sensor of thefirst embodiment;

FIG. 5 is a detailed circuit diagram showing the integrating circuit inthe analog-to-digital converter array of FIG. 4;

FIG. 6 is a block diagram showing a processing element and a shiftregister employed in the high-speed vision sensor of the firstembodiment;

FIG. 7(A) is a circuit diagram showing a register matrix employed in theprocessing element of FIG. 6;

FIG. 7(B) is a control timing chart of the processing element of FIG. 6;

FIG. 8 is a flowchart showing the process of the high-speed visionsensor of the first embodiment;

FIG. 9 is a flowchart of the process S200 in FIG. 8;

FIG. 10 is a timing chart for the transfer and calculating processesaccording to a comparative example;

FIG. 11 is a timing chart for the transfer and calculating processesaccording to the first embodiment;

FIG. 12 is an explanatory diagram showing a high-speed vision sensor forthree-dimensional vision according to a second embodiment of the presentinvention;

FIG. 13 is an entire block diagram showing the high-speed vision sensorof the second embodiment;

FIG. 14(A) is an explanatory diagram showing how left and right camerasare arranged in the second embodiment;

FIG. 14(B) is an explanatory diagram showing the positional relationshipof images taken by the left and right cameras and the object of theimages;

FIG. 15 is a flowchart showing the process of the high-speed visionsensor of the second embodiment;

FIG. 16 is a flowchart showing the process S400 in FIG. 15;

FIG. 17 is a timing chart for the transfer and calculating processes ofa comparative example of a system that has a plurality of cameras;

FIG. 18 is a timing chart for the transfer and calculating processesusing a system with plural cameras according to the second embodiment;

FIG. 19 is an explanatory diagram showing a high-speed vision sensor forcolor image recognition according to a third embodiment of the presentinvention;

FIG. 20 is an entire block diagram showing the high-speed vision sensorof the third embodiment;

FIG. 21 is a flowchart showing the process of the high-speed visionsensor of the third embodiment;

FIG. 22 is a flowchart showing the process S600 in FIG. 21;

FIG. 23 is an explanatory diagram showing the structure of a high-speedvision sensor for recognition of a moving object according to a fourthembodiment of the present invention;

FIG. 24 is a block diagram showing the high-speed vision sensor of thethird embodiment;

FIG. 25 is a flowchart showing the process of the high-speed visionsensor of the fourth embodiment;

FIG. 26 is a flowchart showing the process S800 in FIG. 25;

FIG. 27 is a block diagram showing a high-speed vision sensor accordingto a fifth embodiment of the present invention;

FIG. 28 is a schematic view showing the construction of the fifthembodiment;

FIG. 29 is a block diagram showing a processing element according to thefifth embodiment;

FIG. 30 is a flowchart showing the process of the high-speed visionsensor of the fifth embodiment;

FIG. 31 is a timing chart for the processes of a parallel processingsystem in a comparative example;

FIG. 32 is a timing chart for the processes of the parallel processingsystem according to the sixth embodiment;

FIG. 33 is a block diagram showing a high-speed vision sensor accordingto a sixth embodiment of the present invention;

FIG. 34 is a block diagram showing a processing element according to thesixth embodiment;

FIG. 35 is a timing chart for the processes of the parallel processingsystem according to the sixth embodiment; and

FIG. 36 is a block diagram showing a modification of the high-speedvision sensor of the present invention.

BEST MODE FOR CARRYING OUT THE INVENTION

A high-speed vision sensor according to preferred embodiments of thepresent invention will be described while referring to FIGS. 1-36.

It is noted that the same parts and components are designated by thesame reference numerals to avoid duplicating description.

First, a high-speed vision sensor 1 according to a first embodiment ofthe present invention will be described with reference to FIGS. 1-11.

FIG. 1 is a block diagram showing the high-speed vision sensor 10according to the present embodiment.

First, a brief description will be given for the overall configurationof the high-speed vision sensor 10, with reference to FIG. 1. Thehigh-speed vision sensor 10 of the present embodiment includes aphotodetector array 11, an analog-to-digital converter array 13, aparallel processing system 14, a control circuit 15, and aninstruction/command bus 16.

N1×N2 photodetectors 120 are arranged two-dimensionally (that is, N2rows×N1 columns) in the photodetector array 11. In other words, ahorizontal photodetector row 110 includes N1 photodetectors 120 whichare lined up in the horizontal direction (X direction). Thephotodetector array 11 includes N2 photodetector rows 110 which arearranged vertically (Y direction), that is, orthogonal to the horizontaldirection (X direction).

The analog-to-digital converter array 13 includes N2 analog-to-digitalconverters 210 which are arranged one-dimensionally (vertically in the Ydirection). N2 analog-to-digital converters 210 are arranged inone-to-one correspondence with the N2 photodetector rows 110 in thephotodetector array 11. Each analog-to-digital converter 210 is forsuccessively converting electric charges, outputted from photodetectors120 that belong to a corresponding photodetector row 110, into voltagesignals and then further analog-to-digital converting the voltagesignals into digital signals.

The parallel processing system 14 is provided with a processing elementarray 40. The processing element array 40 includes N1×N2 processingelements (PE) 400 that are arranged two-dimensionally (that is, N2rows×N1 columns) in one-to-one correspondence with the photodetectors120. The processing element array 40 is also provided with N2 shiftregister lines 420 (lines of shift registers for transferring imagedata) in one-to-one correspondence with the N2 rows of processingelements. Each shift register line 420 includes N1 shift registers 410(shift registers for image data transfers). The total number N1 of theshift registers in each line 420 is equal to the total number ofprocessing elements 400 in the corresponding row. The N1 shift registers410 are connected in series in each shift register line 420. Each shiftregister 410 is connected to a corresponding processing element 400 ofthe corresponding row.

The control circuit 15 is for controlling the entire circuit in thehigh-speed vision sensor 10 by transmitting command signals thereto. Theinstruction/command bus 16 is for transferring signals from the controlcircuit 15 to each circuit.

With this construction, N2 data lines are used in the high-speed visionsensor 10 to connect the photodetector array 11 with the parallelprocessing system 14. Accordingly, the photodetector array 11 and theparallel processing system 14 can be formed on separate circuit boards,as shown in FIG. 2, in a configuration that enables the operations ofeach device to be verified separately.

Therefore, production of the high-speed vision sensor 10 can be morestable. Further, by forming the photodetector array 11 and the parallelprocessing system 14 on separate circuit boards in this way, it ispossible to manufacture both devices with a high level of integration.In addition, it is possible to employ a processing method suitable forthe characteristics of each device, thereby enabling a more stableproduction. It is noted that as shown in FIG. 2, the parallel processingsystem 14 is divided into a plurality of regions and mounted on acircuit board. However, such division is not necessary. It is also notedthat because each component of the high-speed vision sensor 10 can beentirely created using the CMOS process, it is possible to create allthe components on one chip, thereby greatly reducing production costs.

Next, the components of each circuit will be described

FIG. 3(A) is a block diagram showing the construction of the controlcircuit 15. The control circuit 15 includes a CPU 150, a memory 151, animage reading control portion 300 (see FIG. 4), an input/outputinterface 152, and the like. All of these components are connected bythe instruction/command bus 16. The input/output interface 152 is forperforming input/output operation with an external device. The memory151 stores therein data of programs to be executed by the CPU 150. Theprograms include: a vision sensor process program (shown in FIG. 8)which will be described later; and an image process program forcontrolling the parallel processing system 14 to execute a parallelimage processing (FIG. 9, for example) during an image processing step(S200 in FIG. 8) of the vision sensor process program. Data of thevision sensor process program and the image process program is writtenin the memory 151 via the I/O interface 152 by an external device 1000(an external computer, for example).

Based on the vision sensor process program stored in the memory 151, theCPU 150 controls the photodetector array 11 and the analog-to-digitalconverter array 13, via the image reading control portion 300, andcontrols the parallel processing system 14. The CPU 150 furthercalculates overall characteristic quantities of an object, being viewed,based on the results of the image processing calculations which areperformed by the parallel processing system 14 during the imageprocessing step (S200). The overall characteristic quantities includethe center of gravity, displacement, and moving speed of the objectbeing viewed. After calculating the overall characteristic quantities,the CPU 150 outputs the calculated results to the external device 1000(for example, an external computer or an external actuator) via the I/Ointerface 152.

FIG. 3(B) is block diagram showing the functions of the control circuit15. The control circuit 15 has a program control portion 15A and acalculating portion 15B according to the function of the CPU 150 thatexecutes the vision sensor process program of FIG. 8. By executing thevision sensor process program of FIG. 8, the program control portion 15Acontrols the photodetector array 11 and the analog-to-digital converterarray 13, via the image reading control portion 300, and also controlsthe parallel processing system 14. More specifically, the programcontrol portion 15A includes an image reading process control portion360 (S101 in FIG. 8) and a parallel process control portion 350(S102-S106 in FIG. 8). The image reading process control portion 360 isfor controlling the image reading control portion 300 (see FIG. 4) topick up images by the photodetector array 11 and the analog-to-digitalconverter array 13. The parallel process control portion 350 is forcontrolling both of: data transfers executed by the shift registers 410in the parallel processing system 14, and calculations executed by theprocessing elements 400, thereby achieving single instruction andmulti-data stream (SIMD) parallel processing. The calculating portion15B performs required calculations, while the program control portion15A is executing the vision sensor process program. The calculatingportion 15B further calculates, based on results obtained by the visionsensor process program, the overall characteristic quantities, such asthe area and the center of gravity, of the picked up images, and makesdeterminations. The calculating portion 15B also communicates with theexternal device 1000, such as an external computer, or controls aseparate external device 1000, such as an external actuator.

Next, the structure of the photodetector array 11 and theanalog-to-digital converter array 13, which serve as an image readingportion, will be described in detail with reference to FIGS. 4 and 5.

The photodetector array 11 functions as a light receiving unit fordetecting light. The analog-to-digital converter array 13 functions as asignal processing unit for converting electric current signals outputtedfrom the photodetector array 11 to voltage signals and further forperforming analog-to-digital conversion onto the voltage signals. Theimage reading control portion 300, in the control circuit 15, isconnected to both of the photodetector array 11 and theanalog-to-digital converter array 13. The image reading control portion300 functions as a timing control unit for transmitting instructionssignals, indicative of operation timings, to both of the photodetectorarray 11 and the analog-to-digital converter array 13.

First, the structure of the photodetector array 11 (light receivingportion) will be described.

As shown in FIG. 4, each photodetector 120 is configured from aphotoelectric conversion element 130 and a switch element 140. Thephotoelectric conversion element 130 is for generating an electriccharge according to the intensity of received light. The switch element140 is connected to a signal output terminal of the photoelectricconversion element 130. The switch element 140 outputs the electriccharge, accumulated in the photoelectric conversion element 130, inresponse to a horizontal scan signal V_(i) (i=1˜N1). Each photodetectorrow 110 is constructed from N1 photodetectors 120 arranged in thehorizontal direction (x-direction) with their corresponding switchelements 140 being connected with one another. The photodetector array11 is constructed from N2 photodetector rows 110 which are arrangedvertically (in the y-direction), that is, perpendicularly to thehorizontal direction. Thus, the photodetectors 120 _(i,j) (i=1˜N1,j=1˜N2) are arranged two-dimensionally in N1 columns×N2 rows.

Next, the structure of the analog-to-digital converter array 13 will bedescribed with reference to the same FIG. 4. The analog-to-digitalconverter array 13 serves as the signal processing unit.

The analog-to-digital converter array 13 is constructed from N2analog-to-digital converters 210 _(j) (j=1˜N2).

Each analog-to-digital converter 210 _(j) is for extracting electriccharges individually from the respective photodetectors in thecorresponding photodetector row 110 _(j) (j=1˜N2), for processing theelectric charges, and for outputting digital signals corresponding tothe magnitudes of the electrical charges.

Each analog-to-digital converter 210 _(j) includes: an integratingcircuit 220 _(j) having a charge amp 221 _(j); a comparator circuit 230_(j); and a capacity control mechanism 240 _(j).

The integrating circuit 220 _(j) includes: the charge amp 221 _(j); avariable capacity unit 222 _(j); and a switch element 223 _(j) Thecharge amp 221 _(j) is for amplifying the charge of an input signalreceived from the photodetector row 110 _(j). The variable capacity unit222 _(j) is connected, on one end, to the input terminal of the chargeamp 221 _(j), and is connected, on the other end, to the output terminalof the same. The switch element 223 _(j) is connected, on one end, tothe input terminal of the charge amp 221 _(j), and is connected, on theother end, to the output terminal of the same. The switch element 223_(j) is turned ON or OFF in response to a reset signal R which issupplied from the image reading control portion 300, thereby togglingthe operation of the integrating circuit 220 _(j) between an integratingstate and a not-integrating state.

FIG. 5 shows a more detailed construction of the integrating circuit220. This drawing shows one example of the integrating circuit that hasan analog-to-digital converting function with a 4-bit (16 gradation)resolution. The integrating circuit 220 will be described below for thiscircuit structure.

The variable capacity unit 222 includes: capacity elements C1-C4; switchelements SW11-SW14; and other switch elements SW21-SW24. Each of thecapacity elements C1-C4 is connected, on one end, to the input terminalof the charge amp 221 that receives signals outputted from thecorresponding photodetector row 110. Each of the switch elementsSW11-SW14 is connected between the other end of a corresponding capacityelement C1-C4 and the output terminal of the charge amp 221. The switchelements SW11-SW14 are for opening and closing in response to capacityinstructions signals C₁₁-C₁₄, respectively. Each of the switch elementsSW21-SW24 has one terminal, which is connected between the correspondingcapacity element C1-C4 and the corresponding switch element SW11-SW14,and another terminal, which is connected to a ground level GND. Theswitch elements SW21-SW24 are for opening and closing in response tocapacity instructions signals C₂₁-C₂₄, respectively. The capacitiesC₁-C₄ of the capacity elements C₁-C₄ have the following relationships:

 C ₁=2C ₂=4C ₃=8C ₄

C ₀ =C ₁ +C ₂ +C ₃ +C ₄

Wherein C₀ is the maximum electrical capacity required by theintegrating circuit 220. It is assumed that the saturation charge of thelight receiving element 130 (see FIG. 4) is Q₀ and that a referencevoltage is V_(REF). In this case, the following relationship issatisfied:

C ₀ =Q ₀ /V _(REF)

The remainder circuits in the analog-to-digital converter 210 _(j) otherthan the integrating circuit 220 _(j) will be described below while onceagain referring to FIG. 4. The comparator circuit 230 _(j) is forcomparing the value of an integral signal V_(s), which is outputted fromthe integrating circuit 220 _(j), with the reference voltage V_(REF),and for outputting a comparison result signal V_(c). Based on the valueof the comparison result signal V_(c), the capacity control mechanism240 _(j) outputs a capacity instruction signal C to notify the variablecapacity unit 222 _(j) in the integrating circuit 220 _(j). The capacitycontrol mechanism 240 _(j) also outputs a digital signal D1 thatcorresponds to the capacity instruction signal C.

It is noted that the analog-to-digital converter array 13 has beendescribed above for the case where the analog-to-digital converter array13 has a 4-bit (16 gradation) resolution. However, the analog-to-digitalconverter array 13 can be configured to have a 6-bit, 8-bit, or otherbit resolution.

The image reading unit 11, 13, having the above-described structure, iscontrolled in image reading timings by the image reading control portion300 in the control circuit 15. As shown in FIG. 4, the image readingcontrol portion 300 includes: a basic timing portion 310, a horizontalshift register 320, and a control signal portion 340. The basic timingportion 310 is for generating a basic timing to attain clock control ofthe entire circuits 11 and 13. The horizontal shift register 320 is forgenerating a horizontal scan signal V_(i) according to a horizontal scaninstruction inputted from the basic timing portion 310. The controlsignal portion 340 is for generating a reset instruction signal R.

Next, the structure of the processing elements 400 and the shiftregisters 410 that make up the parallel processing system 14 will bedescribed.

As described already, as shown in FIG. 1, the parallel processing system14 is provided with N2 shift register lines 420 _(j) (j=1˜N2) incorrespondence with the N2 photodetector rows 110 _(j) (j=1˜N2) in thephotodetector array 11 and in correspondence with the N2analog-to-digital converters 210 _(j) (j=1˜N2) in the analog-to-digitalconverter array 13. In each shift register line 420 _(j), N1 shiftregisters 410 _(i,j) are connected in series. Each shift register 410_(i,j) has a plurality of bits (four (4) bits in this example). As shownin FIG. 6, the parallel process control portion 350, in the controller15, is connected to each shift register 410 _(i,j) (i=1˜N1, j=1˜N2). Theparallel process control portion 350 controls transfer of data from theanalog-to-digital converter array 13 to the required position among theprocessing elements 400 by outputting a transfer start signal to eachshift register 410 _(i,j).

In the parallel processing system 14, the processing elements 400 _(i,j)(i=1˜N1, j=1˜N2) are arranged two-dimensionally in one to onecorrespondence with the N1×N2 photodetectors 120 _(i,j) (i=1˜N1,j=1˜N2). Each processing element 400 _(i,j) (i=1˜N1, j=1˜N2) isconnected to a corresponding shift register 410 _(i,j) (i=1˜N1, j=1˜N2)in a corresponding shift register line 420 _(j) (j=1˜N2). As shown inFIG. 6, the parallel process control portion 350 is connected to eachprocessing element 400 _(i,j). The parallel process control portion 350is for controlling calculations executed by each processing element 400.

Next, the structure of the processing element 400 will be described inmore detail with reference to the block diagram shown in FIG. 6.

The processing elements 400 are configured to perform SIMD-type parallelprocessing. In this type of parallel processing, all the elements arecontrolled by common control signals. Accordingly, the number oftransistors mounted in each element can be reduced, enabling higherintegration in the parallel processing system 14 and enabling anincreased number of elements.

The processing element 400 includes a register matrix 401, an A-latch402, a B-latch 403, and an arithmetic logic unit (ALU) 404. The registermatrix 401 has 4×8 bits, and can be randomly accessed. The registermatrix 401 is of a 1-bit shift type. The register matrix 401 is forperforming data storage operation and data input/output operation. Morespecifically, the register matrix 401 of each processing element 400 isfor receiving, from the corresponding shift register 410, a digitalsignal D1 that is equivalent to a signal outputted from a correspondingphotodetector 120, and for storing it therein. The register matrix 401of each processing element 400 is directly connected to the registermatrices 401 of four neighboring processing elements 400 thereof. Theregister matrix 401 can therefore receive digital signals that arestored in the register matrices 401 of the neighboring processingelements and can store them therein. The ALU 404 is for performing bitserial calculations to perform calculation one bit at a time beginningfrom the least significant bit. The A-latch 402 and the B-latch 403 arefor receiving signals that are stored in the register matrix 401, andfor supplying these signals to the ALU 404 for calculations.

In each processing element 400 having the above-described structure, theA-latch 402 and the B-latch 403 can read data from desired registers inthe register matrix 401. The ALU 404 performs calculations based on thisdata. The results of the calculations are written back to desiredregisters in the register matrix 401. The processing element 400executes various calculations by repeatedly performing this operationcycle.

More specifically, the ALU 404 is a one-bit calculator havingcalculating functions for calculating logical product (AND), logical sum(OR), exclusive or (XOR), addition (ADD), Carry-addition, and the like.The ALU 404 can process only one bit at a time. Therefore, the ALU 404can execute multi-bit calculations by performing a series of one-bitcalculations. It is noted that complex calculations can be described bya combination of plural calculations. Accordingly, the ALU 404 performscomplex calculations by repeatedly performing calculations each timeselecting one function from those available to be performed by the ALU404. For example, multiplication can be described as a combination ofadditions. Therefore, the ALU 404 can perform multiplication byrepeating adding functions a plurality of times. Subtraction can beimplemented by first reversing the bits in the number to be subtracted,adding one (1) to make the value negative, and then performing addingoperation. Division can be implemented by setting the divisor to anumber, such as 8, 4, or 2, and shifting bits. (For example, the bitsare shifted three places to the right if the divisor is 8.) An absolutevalue for a negative number can be calculated by reversing the sign ofthe negative number, whose sign bit is 1, that is, by reversing the bitand then adding one (1) to the result.

As shown in FIG. 7(A), in the register matrix 401, there are arranged:24 one-bit registers 4010 and eight function registers 4012. The 24one-bit registers 4010 can be randomly accessed. The eight functionregisters 4012 are for enabling input/output operations with externaldevices and with neighboring processing elements 400. All of theseregisters are treated as defining a single address space. The numbersappearing in the registers 4010 and 4012 in FIG. 6 indicate theaddresses assigned to the corresponding registers. More specifically,the 24 registers 4010 are assigned addresses 0-23, while the eightfunction registers 4012 are assigned addresses 24-31. With thisconfiguration, input/output data can be accessed in the same manner asreading data from and writing data into the registers.

As shown in FIG. 7(A), the register matrix 401 further includes one ORgate 4014, which is connected to all of these 32 registers 4010, 4012.All of the registers 4010 and 4012 are connected to the A-latch 402 andthe B-latch 403 via the single OR gate 4014. When one address in theregisters 4010 and 4012 is selected, only the selected register outputsits content, while all the remaining non-selected registers output zero(0). The result of the OR function on the values from all the registersis outputted to the A-latch 402 or the B-latch 403 as the output of theentire register matrix 401.

The function registers 4012 are mainly used for performing input andoutput operations. More specifically, the function register 4012 ataddress 30 is connected to a corresponding shift register 410, and isused for inputting data from the shift register 410. The functionregister 4012 at address 24 is used for outputting data to registermatrices 401 of neighboring processing elements 400 that are located onall the four sides of up, down, left, and right. The function registers4012 at addresses 24-27 are used for inputting data from registermatrices 401 of the neighboring processing elements 400 on the foursides, respectively. The function register 4012 at address 28 is usedfor outputting data to the calculating portion 15B. It is noted that thefunction register 4012 at address 28 is configured to always read zero(0), while the function register 4012 at address 29 is configured toalways read one (1).

The parallel process control portion 350 controls accesses to theregister matrix 401 and calculations by the ALU 404, thereby controllingthe entire processes of calculations and input/output operations of theprocessing element 400. In order to write data, inputted from the shiftregister 410, to address zero (0) in the register matrix 401, forexample, the parallel process control portion 350 outputs, to theprocessing element 400, an instruction to calculate the logical OR of 0(register address 28) and the sensor input (register address 30) andthen to write the calculated result to register address (0). It is notedthat as shown in FIG. 7(B), the parallel process control portion 350sets one cycle of operation to the value four times as long as a basicclock CLK. While providing various clocks CLK2, CLK4, the parallelprocess control portion 350 successively performs control operation toread data into A-latch, to read data into B-latch, and to write theresult of calculation into the register matrix. By performing thisprocess repeatedly over a plurality of cycles, it is possible to attainrequired inputting/outputting operation with the register matrix 401 andto attain calculations at the ALU 404.

Next, the operations of the present embodiment will be described withreference to FIGS. 4-6.

First, the operations of the image reading portion 11, 13 will bedescribed.

First, the image reading control portion 300 sets the reset signal R toenabled, and sets all of the switches SW11-SW14 in the variable capacityunit 222 of FIG. 5 to ON and sets all the switches SW21-SW24 to OFF.With this operation, the capacity between the input and output terminalsof the charge amp 221 is set to C₀. At the same time, all the switchelements 140 shown in FIG. 4 are set to OFF, and the horizontal scansignal V_(i) is set to a state that does not select any photodetectors120 _(i,j). From this state, the reset signal R is set to disabled, andeach integrating circuit 220 is controlled to start integrationoperations.

After starting the integration operation, a horizontal scan signal V₁ isoutputted for setting, to ON, only the switch element 140 in the firstphotodetector 120 _(1,j) among the photodetectors in each of the N2photodetector rows 110 _(j). When the switch element is thus turned ON,a charge Q₁, which has been accumulated in the photoelectric conversionelement 130 in response to light received until present, is outputted asa current signal from the photodetector array 11. Thus, a signal is readfrom the photoelectric conversion element. The charge Q₁ then flows tothe variable capacity unit 222 _(j), which is now being set at capacityC₀.

Next, the internal operations of the integrating circuit 220 will bedescribed with reference to FIG. 5. The capacity control mechanism 240(FIG. 4) opens switches SW12-SW14, and then closes the switchesSW22-SW24. As a result, the integral signal V_(s) is outputted to havethe following voltage value:

V _(s) =Q/C ₁

The integral signal V_(s) is inputted into the comparator circuit 230,and is compared with the reference voltage V_(REF). If the differencebetween the integral signal V_(s) and the reference voltage V_(REF) iswithin the range of resolution, that is, lower than or equal to ±C₄/2,then the integral signal V_(s) is determined to match the referencevoltage V_(REF). In this case, no further capacity control is performedand the integral operation ends. On the other hand, if the differencedoes not fall within the range of resolution, then further capacitycontrol will be performed to continue the integral operation.

For example, if V_(s) is greater than V_(REF), the capacity controlmechanism 240 opens the switch SW22 and subsequently closes the switchSW12. As a result, the integral signal V_(s) changes to have thefollowing voltage value:

V_(s) =Q/(C ₁ +C ₂)

This integral signal V_(s) is inputted into the comparator circuit 230(FIG. 4) and is compared with the reference voltage V_(REF).

On the other hand, if V_(s) is smaller than V_(REF), then the capacitycontrol mechanism 240 opens both switches SW11 and SW22 and subsequentlycloses the switches SW12 and SW21. As a result, the integral signalV_(s) changes to have the following voltage value:

V _(s) =Q/C ₂

This integral signal V_(s) is outputted to the comparator circuit 230 tobe compared with the reference voltage V_(REF).

By repeating this feedback loop among the integrating circuit 220, thecomparator circuit 230, the capacity control mechanism 240, and back tothe integrating circuit 220, comparison operation and capacity settingoperation (that is, control of the ON/OFF settings of the switchesSW11-SW14 and SW21-SW24) are repeated until the integral signal V_(s)matches the reference voltage V_(REF) within the resolution range. Thevalues of the capacity instructions signals C_(11-C) ₁₄, which indicatethe ON/OFF states of the switches SW11-SW14 at the time when theintegration operations are ended, constitute a digital signal thatcorresponds to the charge Q₁, wherein the most significant bit (MSB) ofthe digital signal is the value of C₁₁, while the least significant bit(LSB) is the value of C₁₄. The input signal is thus analog-to-digitalconverted into the digital signal, and the digital signal is outputtedas a digital signal D1 to the processing element array 14. As describedabove, the present apparatus successively determines the values of therespective bits in the digital signal D1 one bit at a time from the MSBto the LSB.

Hence, according to the present embodiment, the integral signal V_(s) isrepeatedly compared with the reference voltage V_(REF), while thecapacity elements C1-C4 are successively turned ON one at a time. Thecomparison result is outputted as a digital signal D1. Morespecifically, C1 is first set to ON, thereby making integral signalV_(s) equal to Q/C1. This integral signal V_(s) is compared with thereference voltage V_(REF). A one (1) is outputted as the MSB (mostsignificant bit) if the integral signal V_(s) is larger than thereference voltage V_(REF). A zero (0) is outputted as the MSB (mostsignificant bit) if the integral signal V_(s) is smaller than thereference voltage V_(REF). Next, C2 is set to ON, making the integralsignal V_(s) be equal to Q/(C1+C2) when MSB=1, or the integral signalV_(s) be equal to Q/C2 when MSB=0. This integral signal V_(s) iscompared to the reference voltage V_(REF). A one (1) is outputted as thesecond bit if the integral signal V_(s) is larger than the referencevoltage V_(REF). A zero (0) is outputted as the second bit if theintegral signal V_(s) is smaller than the reference voltage V_(REF).Analog-to-digital conversion is executed by repeating theabove-described process the number of times equal to the number of bitsrequired.

When output of a digital signal equivalent to the photoelectric outputfrom the first photodetector 120 _(1,j) is completed, the reset signal Ris enabled. The reset signal is then disabled again, and the capacityvalue of the variable capacity unit 222 _(j) is initialized.Subsequently, a horizontal scan signal V₂ is outputted for turning ONthe switch element 140 only in the second photodetector 120 _(2,j) ineach row 110 _(j). Then, the above-described process is performed againfor reading the photoelectric output from the second photodetector 120_(2,j) and for outputting a corresponding digital signal. Hereafter, bysuccessively changing the horizontal scan signal V_(i), photoelectricoutputs are read successively from all the photodetectors 120 _(i,j),and corresponding digital signals are outputted to the parallelprocessing system 14.

Next, the operations of the processing elements 400 will be describedwith reference to FIG. 6.

The digital signal of the output from each photodetector 120 _(i,j),which is produced by the analog-to-digital conversion, is transmitted tothe register matrix 401 of a corresponding processing element 400 _(i,j)via the shift registers 410. This transmission process is executed, inthe corresponding shift register line 420, by successively transferringthe signal in one shift register to the shift register 410 of aneighboring pixel. Because the shift registers 410 are provided in thepresent embodiment, signal transmission operation by the shift registerscan be performed independently from the calculation processing performedby the processing elements 400. Accordingly, it is possible to perform apipeline process to control the shift registers 410 to transmit nextdata while controlling the processing elements 400 to performcalculation processing onto present data. It is therefore possible toperform calculations at the processing elements 400 at a higher framerate. It is noted that the shift registers 410 begin transferring theanalog-to-digital converted data in response to a transfer start signal,which is sent from the parallel process control portion 350. The shiftregisters 410 return a “data transfer complete” signal to the parallelprocess control portion 350 after performing bit-shifting transferoperation by the number of bits that is equal to the product of thetotal number (N1) of elements in a row and the number of bits of analogdata levels. Thus, an efficient transmission operation can be attained.Hence, according to the present embodiment, the pipeline process can beexecuted by performing the calculation processes and the transmissionprocesses in parallel. It is possible to reduce the amount of wait timebetween the calculation processes and the transmission processes in thesuccessive frames, thereby achieving faster image processing.

The image process calculations are conducted in the processing elements400 as described below. Signals stored in the register matrices 401 ofthe respective processing elements 400 are transmitted to otherprocessing elements 400 when required. In each processing element,signals for calculations are read from the register matrix 401 into theA-latch 402 and the B-latch 403. Predetermined calculation is achievedin the ALU 404. The calculated result is outputted via the registermatrix 401 to the calculating portion 15B. It is noted that in theparallel processing system 14, the above-described image processingcalculations are performed simultaneously by all the processing elements400 in a parallel processing manner. Accordingly, calculation can beperformed at an extremely high rate of speed. The calculating portion15B calculates required image characteristic values based on the resultsof calculations obtained by the parallel processing system 14, andoutputs the calculated results to the external device 1000. It is notedthat the results of calculations obtained by the parallel processingsystem 14 can be outputted directly to the external computer or to otherexternal devices 1000. For example, the results of calculations obtainedby the parallel processing system 14 can be used as ON/OFF signals foran external equipment.

Next, a series of vision sensor processes performed by the presentvision sensor 10 will be described with reference to FIGS. 8 and 9. Thisseries of processes covers the image input process and the imagetransmission process up through the completion of the calculationprocess.

The CPU 150, in the control circuit 15, controls in S101 the imagereading control portion 300 (see FIG. 4) thereby successively switchingthe reset signal R between the enabled condition and the disabledcondition and successively switching the horizontal scan signal V_(i).As a result, in each row j (j=1, . . . , N2) of the photodetector array11, image data (frame data; hereinafter referred to as I(x, y)), whichis outputted from each photodetector 120 _(i,j) (hereinafter referred toas photodetectors 120 (x, y)), is successively inputted into theparallel processing system 14 via the corresponding analog-to-digitalconverter 210 _(j).

In S102, in each row j, data inputted from the analog-to-digitalconverter 210 _(j) is transmitted in succession via the shift registers410. This transfer process continues in S103 until image data from eachphotodetector 120 reaches the shift register 410 _(i,j) at thecorresponding position (i, j) (hereinafter referred to as (x, y)).

After transmission has been completed, in S104, data I(x, y) of eachpixel is transferred from the corresponding shift register 410 to theregister matrix 401 of the corresponding processing element 400 _(i,j)(hereinafter referred to as processing element 400(x, y)). As shown inFIG. 6, data in the shift register 410 is composed of a plurality ofbits (four bits in the present embodiment) Accordingly, data istransferred from the shift register 410 one bit at a time into theregister matrix 401. In S200, each processing element 400 is controlledto perform necessary image processing.

While each processing element 400 performs calculation processes inS200, each shift register 410 that has completed transferring data tothe corresponding processing element 400 continues to the process of thenext frame in S105, and the processes of S101-S103 are executed again.After completing the calculating process in S200, each processingelement 400 advances to the next frame process in S106. Accordingly,image data for the next frame is transmitted in S104 from each shiftregister 410 to the register matrix 401 of the corresponding processingelement 400. By repeating the above-described processes, each processingelement 400 can perform the calculation process onto the present framedata in S200 while each shift register 410 transfers the next frame datain S101-S103, thereby reducing wasteful wait time.

Next, the operations of the processing elements 400 executed in S200will be described in detail. In this description, the processingelements perform an “edge-enhancement” as an example.

It is noted that “edge-enhancement” is one of the processes which areused most frequently in image processing. In order to calculate theedge-enhancement in one of the simplest manners, calculation isperformed onto every two neighboring pixels. More specifically, thedifference is calculated between the intensity of each pixel and theintensity of a neighboring pixel on the left side thereof. For example,if the input image intensity at a position (x, y) is I(x, y), then imageintensity data I′(x, y) of an edge-enhancement image can be expressed bythe following equation:

I′(x,y)=|I(x,y)−I(x−1,y)|.

In order to perform the edge-enhancement operation, the process shown inFIG. 9 is executed in S200. This image processing operation is performedsimultaneously by each of all the processing elements 400 according to aparallel processing manner.

More specifically, first, in S210, to each processing element (x, y),data I(x−1, y) of its left neighboring pixel is transferred via theinput terminal for its four neighboring pixels. The data I(x−1, y) isthen stored in the register matrix 401 of the subject processing element(x, y). As a result, data I(x, y) and I(x−1, y) is currently stored inthe register matrix 401 of each processing element (x, y). In S211, dataI(x, y) is transferred into the A-latch 402 in order from its leastsignificant bit, and data I(x−1, y) is transferred into the B-latch 403in order from its least significant bit. In S212, the ALU 404 calculatesthe difference between these two values I(x, y) and I(x−1, y). In S213,the result of the calculation is stored temporarily in the registermatrix 401. After this calculation is completed, the result of thecalculation is read again into the A-latch 402 in S214. In S215, the ALU404 calculates the absolute value of the difference. In S216, the resultof this calculation is stored in the register matrix 401. When thecalculation of the absolute value is completed, in S217, the resultpresently stored in the register matrix 401 is outputted to thecalculating portion 15B of the control circuit 15, and this imageprocessing process ends. The above-described calculation operation canbe performed extremely fast because the operation is executed by each ofall the processing elements 400 simultaneously according to the parallelprocessing manner.

When receiving the calculated results, the calculating portion 15Bcalculates, based on the calculated results, the overall characteristicvalues of an imaging object, such as the object's center of gravity, theobject's displacement, and the object's velocity. Based on thecalculated results of the overall characteristic values, the calculatingportion 15B determines the quality of the image. The calculating portion15B may output the result of this determination as a control signal tothe external device 1000, such as an external actuator.

Thus, according to the present sensor 10, the parallel processing system14 performs image processing calculations, such as calculations of thesum, the difference, the product, or the matching of the images. On theother hand, the calculating portion 15B performs calculations of theoverall characteristic quantities of the object, such as the center ofgravity, the displacement, and the velocity of the object.

FIG. 10 is a timing chart of the transfer process and the calculatingprocess according to a comparative example, while FIG. 11 is a timingchart of the transfer process and the calculating process according tothe present embodiment. In the comparative example, the parallelprocessing system 14 is not provided with shift registers 410. Thetransfer process is therefore performed by the processing elements 400themselves. Accordingly, as shown in FIG. 10, the calculating process isconducted after the transfer process is completed. In order to performthe next transfer process, it is necessary to wait until the presentcalculation process has ended. The time required to process one frame isequal to or longer than the sum of the time required for the transferprocess and the time required for the calculating process. In contrast,in the high-speed vision sensor 10 of the present embodiment, a circuitdedicated for the transfer process and a circuit dedicated for thecalculating process are provided separately. The transfer process andthe calculating process are performed independently from each other.Accordingly, as shown in FIG. 11, while the calculating process isperformed with data that has just been transferred by the transferprocess, it is possible to begin the next transfer process. Hence, thetime to process one frame is equal to the longer one of the timerequired for the transfer process and the time required for thecalculating process. Thus, it is possible to speed up the calculatingand transfer processes through a pipeline method.

In the image processing step of S200, each processing element 400,having the above-described configuration, can perform various imageprocessing operation in addition to the above-described edge-enhancementoperation.

For example, when performing a four-neighboring edge-enhancement, if theintensity of an input image at point (x, y) is represented by I(x, y),the image intensity I′(x, y) at point (x, y) of a four-neighboringedge-enhancement image can be represented by the following equation:

I′(x,y)=I(x,y−1)+I(x,y+1) +I(x−1,y)+I(x+1, y)−4I(x,y).

When performing a four-neighboring smoothing, the image intensity valueI′(x, y) of a four-neighboring smoothed image is represented by thefollowing equation:

I′(x,y)=(4I(x,y)+I(x−1,y) +I(x+1,y)+I(x,y−1) +I(x,y+1))/8.

It is noted that Table 1 below shows a list showing the number of stepsrequired for performing each of several algorithms, which include theabove-described algorithms and other algorithms and which are frequentlyused in image processing operations. The Table 1 also shows the lengthof processing time required for completing each algorithm when thesubject algorithm is executed according to the present embodiment.

TABLE 1 Number of Time required Name of Process steps (μs) Two- 1-bit 50.40 neighboring input/ edge output detection Four- 1-bit 11 0.72neighboring input/ edge output detection Four- 1-bit 14 1.0 neighboringinput/ smoothing output Four- 8-bit 70 5.6 neighboring input/ edgeoutput detection Four- 8-bit 96 7.7 neighboring input/ edge outputdetection Four- 1-bit 23 1.9 neighboring input/ thinning output Eight-1-bit 53 4.2 neighboring input/ thinning output Convolution 1-bit 40 3.2input, 4-bit output Convolution 4-bit 372 30 input, 11-bit outputPoisson's 1-bit 63 5.0 equation input, 8-bit output

As can be apparent from Table 1, general image processings, such assmoothing, thinning, convolution, correlation, and masking, can beattained extremely rapidly by executing a complete parallel processing.It is noted that the length of each calculation time listed in Table 1does not include the time required by the shift registers 410 totransfer image data. If no shift registers 410 are provided, an extraamount of time would be required to transfer image data to theprocessing elements 400 in addition to the calculating time. It is alsonoted that the data transfer rate is limited by the conversion speed ofthe analog-to-digital converter 210. If the length of time required foranalog-to-digital conversion is 1 μs/bit, for example, then the time,required for transferring 128×128 pixels×8 bits' worth of image data inparallel using the 128 lines, would be 128 pixels×8 bits×1 μs/bit=1 ms.Hence, the transfer process would require several times as long as theamount of time required to perform the calculations, making difficultimage processing in real-time. Contrarily, according to the presentembodiment, the transfer processes and the image processes are executedin parallel, and therefore it becomes possible to process such imagedata within approximately 1 ms. Accordingly, the present embodiment canbe applied to such fields, as factory automation (FA) robot control,contrary to conventional vision sensor devices which have slow imageprocessing speeds and slow transfer rates.

The object of the present embodiment is to provide an image processingsystem of a practically high speed and sufficiently high resolution. Itis noted that the robot control in FA systems requires such a resolutionthat 128×128 photodetectors 120 or more are arranged. According to thepresent embodiment, the photodetector array 11 and the parallelprocessing system 14 can be separated from each other. Accordingly, eachdevice can be constructed with a high level of integration. It istherefore possible to attain the required high resolution. Additionally,the processing speed has to match the actuator speed of a robot (1-10ms). According to the present embodiment, the processing speed isdetermined by the speed of the analog-to-digital conversion performed inthe analog-to-digital converters 210. This conversion rate can be madesufficiently fast as described below.

For example, according to the present embodiment, the analog-to-digitalconversion rate per pixel is 1 μs/bit. For example, when convertinganalog input signals to 6-bit digital data (64 gradation), the entirelength of time required to convert signals, outputted from 128photodetectors 120 in one row, to the digital signals is 0.768 ms=6μs×128. On the other hand, the processing elements are provided inone-to-one correspondence with the photodetector elements. All theprocessing elements are operated in parallel. Accordingly, most imageprocessings can be completed within 0.4 ms as apparent from Table 1.Additionally, by performing the calculating processes and the transferprocesses in parallel, it is possible to reduce the amount of dead timebetween the respective processes and to shorten the overall processingtime.

As described already, each analog-to-digital converter 210 performsanalog-to-digital conversion beginning from the most significant bit.Accordingly, it is possible to change the gradation of theanalog-to-digital conversion, by outputting a reset signal R after adesired number of bits have been converted so that the process willshift to perform an analog-to-digital conversion of the next opticalsignal. It is therefore possible to perform complicated processes at ahigher rate of speed. For example, in order to track an object which ismoving at a high rate of speed, it is preferable to process images intwo levels, that is, one bit. In this case, the transfer time can beshortened to 0.128 ms, which is about ⅙ the time required fortransferring six bits. This method can therefore be applied to thecontrol of high-speed feedback operation. Conversely, when the object ismoving slowly, the gradation can be raised to be able to track theobject with a higher precision.

In order to allow the output from the analog-to-digital converter tohave a variable bit length, it is necessary to adjust, before inputtingthe output to the shift registers, the varied bit length of input datato a fixed length. It is noted that a shift register line, fortransferring data at a normal length of 8 bits, for example, isconstructed from one set of shift registers that has the total fixedlength equal to the product of eight bits and the total number (N1) ofpixels in one row. In the set of shift registers, a plurality of eightbit-shift registers, which are spaced from one another and each of whichhas eight bits, function as individual shift registers to transfer datafor their corresponding pixels. Therefore, if the bit length of theimage data for each pixel is not fixed to eight bits, then it isimpossible to transfer each pixel's worth of image data correctly to theshift register at the corresponding position. For this reason, eachpixel's worth of image data is added with a dummy signal to have eightbits in total before the image data is transferred to the shiftregister. It can be ensured that image data will be transferredcorrectly.

As described above, the high-speed vision sensor 10 of the presentembodiment includes the analog-to-digital converter array 13 and theparallel processing system 14. The analog-to-digital converter array 13has one analog-to-digital converter 210 in correspondence with all thephotodetectors 120 that belong to one row of the photodetector array 11.In the parallel processing system 14, the processing elements 400 areprovided in one to one correspondence with the photodetectors 120, andthe shift registers 410 are provided also in one to one correspondencewith the photodetectors 120. By providing a processing element 400 incorrespondence with each photodetector 120, it is possible to performhigh-speed image processing calculations between neighboring pixels byparallel processing. Further, by providing a shift register 410 incorrespondence with each processing element 400, the transfer processcan be performed independently from the calculation processing, therebyachieving calculation and transfer processes efficiently. Because thecalculation process and the transfer process are executed in parallel,it is possible to reduce wait time between the respective processes andto perform image processing at a faster overall speed.

Additionally, by providing an analog-to-digital converter 210 in eachrow, it is possible to reduce the number of transfer lines, providedbetween the photodetectors 120 and the processing elements 400, incomparison to a configuration where an analog-to-digital converter 210is provided in correspondence with each photodetector 120. Thephotodetectors 120 and the processing elements 400 can be producedseparately, before being assembled together. Accordingly, bothcomponents can be manufactured with an optimal level of integration.Production of the high-speed vision sensor 10 having a large number ofpixels can be simplified. It is noted that because one analog-to-digitalconverter 210 is provided for each row, the overall processing speed islimited by the A/D conversion process. However, most processes can becompleted within one millisecond, even when 128×128 pixel images, whichare sufficient for FA robot control, are processed at a 64 gradation.Such a high-speed processing can be attained. Accordingly, thehigh-speed vision sensor 10 of the present embodiment not only can beconstructed with a simple circuit, but also can perform efficienthigh-speed calculations even onto a plurality of images.

Next, a second embodiment of the present invention will be describedwith reference to FIGS. 12-18.

In the first embodiment, only one photodetector array 10 is provided.Contrarily, according to the second embodiment, a plurality ofphotodetector arrays 10 are provided. Signals outputted from theplurality of photodetector arrays 10 are inputted into the singleparallel processing system 14, and are subjected to calculationprocessing.

FIG. 12 is an explanatory diagram showing the general construction ofthe high-speed vision sensor 10 according to the present embodiment. Thehigh-speed vision sensor 10 of the present embodiment includes twophotodetector arrays for performing three-dimensional imaging operation.FIG. 13 is a block diagram showing a more detailed construction of thehigh-speed vision sensor 10.

As shown in FIG. 12, the image of an object 2000 a is guided tophotodetector arrays 11L and 11R via lenses 501L and 501R, and areprocessed by the single parallel processing system 14. The lenses 501Land 501R are disposed in different positions from each other. Thephotodetector array 11L corresponds to the left eye, and photodetectorarray 11R corresponds to the right eye. The photodetector arrays 11L and11R are arranged so that they will transfer data in the same direction,as indicated by the arrows T in the diagram, and so that their pixels,which correspond to each other along the data transfer direction, aredisposed as being shifted from each other by an optical base length b inthe horizontal direction. More specifically, as shown in FIG. 13, thephotodetector arrays 11L and 11R are arranged so that all thephotodetector rows 110 _(j) in both photodetector arrays extend in thesame horizontal direction.

It is noted that the world coordinate system (x, y, z) is defined asshown in FIG. 14(A). In this world coordinate system, the right eye lens501R is disposed in front of the photodetector array 11R. The right eyelens 501R and the photodetector array 11R construct a right eye camera500R. Similarly, the left eye lens 501L is disposed in front of thephotodetector array 11L. The left eye lens 501L and the photodetectorarray 11L construct a left eye camera 500L. Both lenses 501L and 501Rhave the same amount of focal length f, and are positioned on the x-yplane such that their optical axes (z-direction) extend parallel to eachother. It is noted that the center of the lens 501L is set as the originin the world coordinate system (x, y, z). The lenses 501L and 501R areshifted from each other by the optical base length b in the x-direction.The photodetector arrays 11L and 11R are located behind thecorresponding lenses 501L and 501R and are separated from thecorresponding lenses 501L and 501R exactly by the distance of the focallength f. Accordingly, the photodetector arrays 11L and 11R are alsopositioned parallel to the x-y plane and are shifted from each other bythe optical base length b in the x-direction. The transfer direction Tof the photodetector arrays 11L and 11R is the same,with thex-direction.

As shown in FIG. 12, the control circuit 15 is provided with a pixelshift amount storage device 15C. The pixel shift amount storage device15C is for storing data of a positional shift amount J between imageswhich are obtained by the photodetector arrays 11L and 11R for oneframe. The positional shift amount J is a result calculated by theparallel processing system 14.

As shown in FIG. 13, each of the photodetector arrays 11L and 11R hasthe same configuration as the photodetector array 11 in the firstembodiment. Similarly, each of analog-to-digital converter arrays 13Land 13R has the same construction as the analog-to-digital converterarray 13 in the first embodiment.

As in the first embodiment, the parallel processing system 14 isprovided with N1 columns×N2 rows of processing elements 400 _(i,j)(i=1˜N1, j=1˜N2) which are arranged two-dimensionally with a one-on-onecorrespondence to the photodetectors 120 _(i,j) in each of thephotodetector arrays 11L and 11R. The parallel processing system 14 isfurther provided with: N2 shift register lines 420R_(j) for transferringdata for the right eye, and N2 shift register lines 420L_(j) fortransferring data for the left eye. The N2 shift register lines 420R_(j)(j=1˜N2) are provided in one-on-one correspondence with the N2 rows ofprocessing elements j (j=1˜N2). The N2 shift register lines 420L_(j)(j=1˜N2) are provided also in one-on-one correspondence with the N2 rowsof processing elements j (j=1˜N2).

The N2 shift register lines 420R_(j) are respectively connected to theN2 analog-to-digital converter 210 _(j) in the analog-to-digitalconverter array 13R. The N2 shift register lines 420L_(j) arerespectively connected to the N2 analog-to-digital converter 210 _(j) inthe analog-to-digital converter array 13L.

In each shift register line 420R_(j), N1 shift registers 410R_(i,j)(i=1˜N1), for transferring data for the right eye, are connected inseries, similarly to the shift register line 410 of the firstembodiment. Each shift register 410R_(i,j) has a plurality of bits (fourbits in the present embodiment). Similarly, in each shift register line420L_(j), N1 shift registers 410L_(i,j) (i=1˜N1), for transferring datafor the left eye, are connected in series. Each shift register410L_(i,j) also has a plurality of bits (four bits in this case). Eachprocessing element 400 _(i,j) (i=1˜N1, j=1˜N2) is connected to both ofthe corresponding shift register 410R_(i,j)(i=1˜N1, j=1˜N2) and thecorresponding shift register 410L_(i,j)(i=1˜N1, j=1˜N2).

In this way, in the parallel processing system 14, the shift registers410R are provided in one-to-one correspondence with the photodetectors120R in the photodetector array 11R, and the shift registers 410L areprovided in one-to-one correspondence with the photodetectors 120L inthe photodetector array 11L. One processing element 400 is provided incorresponding to each pair of photodetectors 120R and 120L. Accordingly,a pair of shift registers 410L and 410R are provided for each processingelement 400. The construction of the control circuit 15 and theinstruction/command bus 16 are basically the same as those described inthe first embodiment shown in FIG. 1, except that the pixel shift amountstorage device 15C is added to the control circuit 15, and that thecontrol circuit 15 controls transfer operations of both shift registers410R and 410L.

In the present embodiment, image signals obtained by the photodetectors120R in the photodetector array 11R are transferred to the shiftregisters 410R via the analog-to-digital converters 210R. Similarly,image signals obtained by the photodetectors 120L are transferred to theshift registers 410L via the analog-to-digital converters 210L. Thesetransfer operations are performed independently from each other. Whenthe image signals for the left and right eyes are transferred from theshift registers 410L and 410R to the processing elements 400, they areprocessed by the processing elements 400. The processing elements 400perform predetermined calculations based on positional information ofthe photodetectors 120R and 120L and the like, thereby attainingthree-dimensional measurements of the object 2000 a.

It is noted that three-dimensional measurements by stereo vision areattained by performing calculations based on image matching and on theprinciple of trigonometry as disclosed in “Stereo vision” written byMasatoshi Okutomi (“Computer Vision,” edited by Matsuyama, et al., NewTechnology Communications, June 1998, pp. 123-124).

Next, a brief description will be given how to calculate the distance toan object using the three-dimensional vision system.

First, an image matching process is conducted to find a matching pointin the right eye image PR and the left eye image PL. The world imagecoordinate system (x_(R), y_(R)) for the right eye image P_(R) and theworld image coordinate system (x_(L), y_(L)) for the left eye imageP_(L) are determined as shown in FIG. 14(B) with respect to the worldcoordinate system (x, y, z) shown in FIG. 14(A). It is assumed that theposition of the object 2000 a is set as shown in FIG. 14(B) with respectto the world coordinate system (x, y, z). Next, a predetermined smallregion (x_(R)±wx_(R), y_(R)±wy_(R)) is set on the right eye image P_(R)at its area centered on the point (x_(R), y_(R)) represented in theworld coordinate system for the right eye image P_(R). A small region ofthe same size with that of the small region (x_(R)±wx_(R), y_(R)±wy_(R))is then set on the left eye image P_(L), and the difference between bothimages is calculated. By conducting this calculation, the center point(x_(L), y_(L)), in the world coordinate system, of an area that causesthe smallest amount of difference is searched. The point (x_(L), y_(L))found by the searching operation is the matching point of the left eyeimage P_(L) with respect to the fixed position (x_(R), y_(R)) in theright eye image P_(R). The positional difference between those areas,that is, the distance (x_(R)−x_(L), y_(R)−y_(L)) is called the parallaxd. It is noted that there occurs little positional difference betweenthe right and left eye images in the direction perpendicular to thedirection, in which the left and right cameras are shifted from eachother. According to the present embodiment, the left and right camerasare placed at positions shifted from each other in the x-direction.Accordingly, the value y_(R) is nearly equal to the value y_(L). Theparallax d is therefore nearly equal to x_(R)−x_(L). In this case, apoint P (x, y, z) in the space is projected onto the left eye image at(x_(L), y_(L)) and onto the right eye image at (x_(R), y_(R))Accordingly, the following formula is established:

Z=bf/d

wherein f is the focal length, b is the length of the baseline (distancebetween the left eye camera 500L and the right eye camera 500R), and dis the parallax (x_(R)−x_(L)). Under the condition where f and b areknown, it is possible to reproduce the three-dimensional position of theimaged object.

Thus, it is possible to find a distance Z to the object by Z=bf/d when fis the focal length of both cameras and when b is the distance betweenthe optical centers of both cameras.

In order to improve the processing speed of three-dimensionalmeasurements, it is necessary to effectively find a matching pointbetween the images taken by the left eye camera 500L and taken by theright eye camera 500R. According to the present embodiment, as shown inFIG. 12, the photodetector arrays 11L and 11R are located so that eachpixel, that is, each photodetector 120 in the photodetector array 11R isarranged as being shifted from a corresponding pixel, that is, acorresponding photodetector 120 in the photodetector array 11L, alongthe direction the same as the data transfer direction T (left-rightdirection or row-extending direction). As a result, the position of aspecific image on a screen in the image taken by the photodetector array11L is shifted from the position of the subject image on a screen in theimage taken by the photodetector array 11R in the left-to-rightdirection. It is therefore possible to detect the position of the objectusing trigonometry in the following manner: The difference between thetwo images is repeatedly calculated while shifting one of the two imagesin the left or right direction. When the one image is shifted by somepixel shift amount to reach some position, the difference between theimages becomes smallest. This indicates that the two images match witheach other. The position of the object can be determined usingtrigonometry based on the pixel shift amount that allows the two imagesto match with each other.

According to the present embodiment, the shift registers 410L and 410Rtransfer the left and right images independently from each other.Accordingly, while performing the calculation of the image difference,it is possible to shift one image for the next calculation, therebyperforming effective processings.

The positional shift amount J calculated in the previous frame is storedin the storage device 15C. If a positional shift amount of fifteen (15)pixels, for example, is determined between the left and right images inthe previous frame, it can be estimated that a positional shift amountof approximately fifteen (15) pixels will be determined in the currentframe. Accordingly, it is preferable to begin, for the current frame,calculation of the image difference from that range. In order to cause arequired amount of pixel shift between the left and right images, one ofthe following methods can be employed: (1) vary the timings between theleft and right cameras; (2) provide a buffer between the cameras and theparallel processing system; and (3) control the transfer registers toperform transfer operation in a state that pixel shift occurs only inone image. It becomes possible to begin a calculation, in the currentframe, from a state where matching occurs in the previous frame.Accordingly, it is possible to reduce the length of time required todetermine the accurate pixel shift amount in the current frame.

Next, one example of the operations of the high-speed vision sensor 10according to the present embodiment will be described with reference toflowcharts of FIGS. 15 and 16. In this example, in order to cause apixel shift between the left and right images with an amount that is thesame as that detected in the previous frame, the method (3) describedabove is employed, wherein the transfer registers are controlled totransfer data such that the pixel shift occurs only in one image.

First, in S301, the photodetector arrays 11L and 11R are controlled topick up left and right images L and R. These images are converted intodigital data by the corresponding analog-to-digital converter arrays 13Land 13R. In S302, left image data L is transferred using the shiftregisters 410L up to a corresponding position (x, y). More specifically,the shift registers 410L are controlled by a transfer start signal tostart transferring left image data. The shift registers 410L arecontrolled to perform transferring operation by shifting bits by thenumber that is equal to the product of the total number (N1) of elementsin a row and the number of bits of analog data levels. At the same time,the shift registers 410R are controlled to transfer right image data Rto a position (x−J, y) that is shifted by a pixel shift amount J, whichhas been obtained in the previous frame and which is being stored in thestorage device 15C. More specifically, the shift registers 410R arecontrolled by the transfer start signal to start transferring data forthe right image. The shift registers 410R are controlled to performtransferring operation by shifting bits by the number that is equal tothe product of (the total number (N1) of elements in a row—J) and thenumber of bits of analog data levels.

Next, in S303, left and right image data L (x, y) and R (x−J, y) istransferred from the shift registers 410L and 410R to the registermatrix 401 at the corresponding position (x, y).

Next, in S400, a process for matching left and right images isperformed.

During the left and right image matching process, as shown in FIG. 16, jis first initialized to J in S410.

Subsequently, a block, having 3×3 pixels or 8×8 pixels, for example, isset in the left image as a reference image. In order to find whichportion of the right image best matches the reference image, acorrelation value C(j), wherein j is initially set to J, is calculatedbetween the block (reference image) in the left image and apresently-corresponding block in the right image.

Here, the correlation value C(j) is expressed by the following equation:

C(j)=Σ|L(x,y)−R(x−j,y)|

wherein x, y indicates all the pixels located in the present block.

More specifically, in S411, the processing element 400 at each pixel (x,y) first calculates the difference, between the currently-stored imagedata L(x, y) and R(x−j, y) in the same manner as in the processes ofS211-S213 in FIG. 9. The processing element in each pixel thencalculates the absolute value of the difference (hereinafter referred toas D(x, y)) in the same manner as in the processes S214-S216 in FIG. 9.The total sum of the absolute values over the entire block is thencalculated to obtain the correlation value C(j). In order to calculatethe total sum, each processing element 400 repeats data transferoperations (to transfer data between neighboring register matrices 401)and repeats data adding functions.

More specifically, in order to calculate the total sum of the absolutevalues over the entire block, if the block is a 3×3 pixel block (whichhas the center pixel and its eight neighboring pixels), for example, thedifference absolute value D(x−1, y) obtained by the pixel in the leftcolumn is first transferred to calculate D(x−1, y)+D(x, y) Thiscalculation is performed by each of all the processing elements 400using SIMD. Next, the difference absolute value D(x+1, y) obtained bythe pixel in the right column is transferred to calculate D(x−1, y)+D(x, y)+D(x+1, y). Next, data obtained by the pixel on the upper row istransferred to obtain the following sum: D(x+1, y−1)+D(x, y−1)+D(x+1,y−1)+D(x+1, y) +D(x, y)+D(x+1, y). Next, data obtained by the pixel onthe lower row is transferred to calculate the following sum: D(x+1,y−1)+D(x, y−1)+D(x+1, y−1)+D(x+1, y)+D(x, y) +D(x+1, y)+D(x+1, y+1)+D(x, y+1)+D(x+1, y+1). Thus, the correlation value C(j) is finallyobtained. When a 8×8 pixel block is used, the above-described total sumoperation further continues while transferring data in the upper, lower,left, and right directions to finally obtain the correlation value C(j).

The obtained value C(j) is transferred to the calculating portion 15B,and is stored therein.

Next, j is changed within the entire search range (No in S412 and S413).More specifically, data R for the right image is transferred to theright or left between the register matrices 401 of the neighboringprocessing elements 400.

Next, the correlation value C(j) is calculated again in S411. Thus,calculation of the correlation value C(j) is repeated within the entiresearch range while the right image data R is transferred between theneighboring processing elements 400 (no in S412 and in S413). It isnoted that this search range is determined dependently on conditionssuch as an estimated maximum speed of the object 2000 a, reduction powerof the lenses 501R and 501L, and pixel pitch in the photodetector arrays11. However, because the frame rate is high, the search range can be setas small as ±1 or ±2, for example, with respect to the initial value Jin the left-right direction.

When all the correlation values C(j) are obtained for the entire searchrange (Yes in S412), the process proceeds to S414. At this time,correlation values C(j), where j=J J−1, J+1, for example, are stored inthe calculating portion 15B. Accordingly, in S414, among all thecorrelation values C(j) in the calculating portion 15B, the smallestvalue C(j) is selected, and the value j for the smallest value C(j) isdetermined. This value j indicates the matching position of the leftimage block with respect to the right image block.

Thus, the process of S400 ends.

Next, as shown in FIG. 15, in S305, the value j determined during theleft and right image matching process of S400 is set as the pixel shiftamount J for the current frame, and is stored in the storage device 15C.Next, in S306, the distance Z=bf/J to the object, appearing in the leftblock image, is calculated, wherein b is the distance between the leftand right cameras, and f is the focal length of the cameras.

It is noted that during the right and left image matching process ofS400, the processes of S411-S413 are executed by the program controlportion 15A controlling the processing elements 400 to perform parallelprocessing. Calculations in steps S414, S305, and S306 are executed bythe calculating portion 15B. It is noted that by performing the matchingprocess of S400 for all the blocks in the light-receiving surfaces 11Land 11R, it is possible to determine the distances to all the objects,which appear in the respective block images, through steps S305 andS306.

The shift registers 410R and 410L perform transfer operation to transferdata of one frame to the processing elements 400. While the processingelements 400 and the calculating portion 15B perform calculations inS400-S306 for the received frame data, the shift registers 410R and 410Lmove on to the process for the next frame in S304 and execute processesof S301-S302. When completing the calculations of S400-S306, theprocessing elements 400 and the calculating portion 15B also move on tothe process of the next frame in S307. Then, in S303, left and rightimage data from the next frame is transferred from the shift registers410L and 410R to the register matrices 401 of the processing elements400. By repeating the above-described procedures, the processingelements 400 and the calculating portion 15B can perform calculationprocesses (S400-S306) onto the present frame while the shift registers410L and 410R transfer left and right image data for the next frame(S301-S302), thereby reducing wasteful wait time.

As described above, according to the sensor 10 of the presentembodiment, the pixel shift amount J, detected in the previous frame, isstored. Accordingly, the search of the matching position for the presentframe can be started from the position determined based on the pixelshift amount detected in the previous frame. Accordingly, the size ofthe search range can be reduced. It is possible to perform searchingoperation efficiently. Especially, according to the high-speed visionsensor 10 of the present embodiment, matching can be performed at anextremely fast frame rate (such as 1 ms). Accordingly, there is almostno movement in the object between successive frames. For example, anobject moving at a rate of 150 kilometers per hour will not move morethan 4 cm between frames. Hence, by knowing in advance the approximatevalue of the pixel shift amount between the left and right images, theobject can be found by searching only several points to the left orright of the known position.

If the shift registers 401R or 410L are not used, it requires time whenthe processing elements 400 transfer image data from both of thephotodetector arrays 11R and 11L. In this case, the time required forthe transfer process is more than double that required for only onephotodetector array. It becomes difficult to perform high-speed imageprocessing. Contrarily, according to the present embodiment, the shiftregisters 410L and 410R perform transfer operations in parallel whilethe processing elements 400 are conducting calculations. Accordingly,high-speed image processing can be attained similarly to the case whereonly one photodetector array is used. It is noted that the number of thephotodetector arrays is not limited to two, but can also be three ormore. In the latter example, it is necessary to provide three or moreshift registers for each processing element, each shift register beingdedicated for the corresponding photodetector array. Using three or morephotodetector arrays can increase precision and can reduce blind spots.

FIG. 17 is a time chart of the transfer process and calculation processin a comparative example, wherein three photodetector arrays (cameras)11 are employed, but no transfer shift registers 410 are employed. FIG.18 is a time chart of the transfer and calculation processes accordingto the present embodiment, wherein three photodetector arrays (cameras)11 are employed. In the comparative example of FIG. 17, the processingelements transfer image signals. Accordingly, the processing elementshave to perform calculation processes after completing transfer of imagesignals. In other words, (a) image signals from camera 1 aretransferred, after which (b) image signals from camera 2 aretransferred, after which (c) image signals from camera 3 aretransferred. After all the transfer operations are completed, (d) thecalculation process is performed. Image signals for the next framecannot be transferred until this calculation process is completed.Accordingly, the time required to process one frame is (the timerequired for transferring signals from one camera)×3+(the time requiredfor the calculation process) or more.

On the other hand, according to the present embodiment, each processingelement is provided with transfer shift registers, the number of whichcorresponds to the number of the cameras (three in this case).Accordingly, the transfer processes for the image signals from therespective cameras can be performed simultaneously, and the calculationprocesses can be performed independently from the transfer processes. Asshown in FIGS. 18(a)-(c), image signals are transferred simultaneouslyfrom the cameras 1-3 by the shift registers. The transferred data isprocessed by the processing elements. While this calculation process isbeing performed, the shift registers move on to transferring image datafor the next frame. Accordingly, the processing elements can perform acalculation process of the next frame immediately after finishing thecurrent calculation process without waiting. Hence, the time required toprocess one frame is determined by the longer of the <time required totransfer signals from one camera> or <time required for the calculationprocess>. According to the present embodiment, even when employing alarge number of cameras, it is still possible to prevent increase inprocessing time.

Next, another embodiment (third embodiment) that employs a plurality ofphotodetector arrays will be described with reference to FIGS. 19-22.

FIG. 19 is an explanatory diagram showing the structure of thehigh-speed vision sensor 10 for color image recognition according to thepresent embodiment. FIG. 20 is a block diagram showing a more detailedstructure of the high-speed vision sensor 10.

An image of an object 2000 b passes through a lens 501 and is separatedinto three RGB colors by a cross dichroic prism 502. Unnecessarywavelength signals are removed by filters 503 r, 503 g, and 503 b forthe three colors. The remaining signals are fed to the photodetectorarrays 11 r, 11 g, and 11 b for the three colors.

As shown in FIG. 20, similarly to the second embodiment, thephotodetector arrays 11 r, 11 g, and 11 b have the same construction asthe photodetector array 11 in the first embodiment. Analog-to-digitalconverter arrays 13 r, 13 g, and 13 b have the same construction as theanalog-to-digital converter array 13 in the first embodiment. As in thesecond embodiment, the parallel processing system 14 includes N1columns×N2 rows of processing elements 400 which are arrangedtwo-dimensionally to correspond one-on-one to the N1×N2 photodetectors120 in each of the photodetector arrays 11 r, 11 g, and 11 b. Thehigh-speed vision sensor 10 of the present embodiment is furtherprovided with N2 shift register lines 420 r in one to one correspondencewith the N2 rows of processing elements. The high-speed vision sensor 10of the present embodiment is further provided with N2 shift registerlines 420 g in one to one correspondence with the N2 rows of processingelements. The high-speed vision sensor 10 of the present embodiment isfurther provided with N2 shift register lines 420 b in one to onecorrespondence with the N2 rows of processing elements. The shiftregister lines 420 r are for transferring red data. The shift registerlines 420 g are for transferring green data. The shift register lines420 b are for transferring blue data. The shift register lines 420 r,420 g, and 420 b are connected to the analog-to-digital converter arrays13 r, 13 g, and 13 b, respectively. As in the second embodiment, N1shift registers 410 r, each having a plurality of bits (four bits inthis example), are connected in series in each shift register line 420r; N1 shift registers 410 g, each having a plurality of bits (four bitsin this example), are connected in series in each shift register line420 g; and N1 shift registers 410 b, each having a plurality of bits(four bits in this example), are connected in series in each shiftregister line 420 b. Each processing element 400 is connected to all ofa corresponding shift register 410 r, a corresponding shift register 410g, and a corresponding shift register 410 b.

Image signals obtained by the photodetector arrays 11 r, 11 g, and 11 bare converted to digital signals by the corresponding analog-to-digitalconverter arrays 13 r, 13 g, and 13 b. Subsequently, the digital signalsare fed to the single parallel processing system 14 and subjected tocalculation processing. In the parallel processing system 14, the shiftregisters 410 r, 410 g, and 410 b are dedicated to transferring imagesignals obtained by the corresponding photodetector arrays 11 r, 11 g,and 11 b to the processing elements 400. With this configuration, aswith the three-dimensional vision system of the second embodiment, imagedata from the respective photodetector arrays 11 can be transferredindependently from and in parallel with the calculation process, therebyachieving high-speed image processing.

According to the present embodiment, digital signals which have alreadybeen separated into RGB colors are transmitted to the processingelements 400. Accordingly, the processing elements 400 can easilyperform high-speed image processing such as color correction and colorconversion. Further, by providing the high-speed vision sensor 10 withadditional plural photodetector arrays 11, each of which has a filter503 for acquiring images of a wavelength region different from the RGBcolors, the high-speed vision sensor 10 can be used for analyzingwavelength characteristics. The color image recognition of the presentembodiment can be used for detecting objects with color information.Some applications of the high-speed vision sensor include real-timequality checking of printed materials and automated vehicle driving bydetection of a white line on a road. According to the presentembodiment, it is also possible to immediately determine whether thecurrently-inputted image is the same as a predetermined template image.

Next, the operations for recognizing color images according to thepresent embodiment will be described with reference to the flowcharts inFIGS. 21 and 22.

First, in S501, RGB image data r(x, y), g(x, y), and b(x, y) is pickedup at each position (x, y) in the corresponding photodetector arrays 11r, 11 g, and 11 b, and is converted into digital signals by thecorresponding analog-to-digital converter arrays 13 r, 13 g, and 13 b.Next, in S502 and S503, RGB image data r(x, y), g(x, y), and b(x, y) istransferred using the corresponding shift registers 410 r, 410 g, and410 b up to the corresponding position (x, y) in the parallel processingsystem 14. In other words, the shift registers 410 r, 410 g, and 410 bare controlled by a transfer start signal to start a transfer ofcorresponding color image data. The shift registers 410 r, 410 g, and410 b perform their transfer operations by shifting bits by the numberthat is equal to the product of the total number N1 of elements in a rowand the number of bits of analog data levels. Next, in S504, RGB imagedata r(x, y), g(x, y), and b(x, y) is transferred, at the transferposition (x, y), to the register matrix 401 of the correspondingprocessing element 400.

Next, the RGB template matching process is executed in S600.

It is noted that reference image data Tr(x, y), Tg(x, y), and Tb(x, y)for RGB colors are prestored in the register matrix 401 of thecorresponding pixel (x, y). This reference image data indicates a frameimage (template) obtained by picking up an image of an ideal object(such as printed material). In the RGB template matching process, inputimage data is compared with the reference image data to find a matchingposition.

More specifically, during the RGB template matching process of S600, asshown in FIG. 22, a correlation value C(m, n), where both of m and n arefirst initialized to zero (0), is first calculated in S610 for each RGBcomponent with respect to one block (3×3 pixel block, for example) inthe reference image and the corresponding current block in the inputimage. The correlation values Cr(m, n), Cg(m, n), and Cb(m, n) for theRGB components are expressed by the following equations:

Cr(m,n)=Σ|r(x,y) −Tr(x+m,y+n)|

Cg(m,n)=Σ|g(x,y) −Tg(x+m,y+n)|

Cb(m,n)=Σ|b(x,y) −Tb(x+m,y+n)|

Wherein x, y represents all the pixels located in the present block. Thecorrelation values are calculated by the processing elements in the samemanner that the correlation value C(j) is calculated in the secondembodiment.

In S610, calculation is further performed to obtain the total sum “Call(m, n)” of the correlation values for all the RGB components. The totalsum Call (m, n) is therefore expressed by the following equation:

Call(m,n)=Cr(m,n) +Cg(m,n)+Cb(m,n)

The total sum Call (m, n) of correlation values is calculated also bythe processing elements 400. The calculated value Call (m, n) is thentransferred to and stored in the calculating portion 15B.

Next, the value of m or n is changed within the entire search range (Noin S611, S612). More specifically, the reference image data Tr, Tg, andTb is transferred between register matrices 401 of neighboringprocessing elements 400 in the up, down, left, or right direction.

Next, in S610, the value Call (m, n) is calculated again. In this way,while transferring reference images Tr, Tg, and Tb between neighboringprocessing elements 400 (no in S611 and S612), calculation of the valueCall (m, n) is repeated over the entire search range. It is noted thatthe search range is determined dependently on conditions such as anestimated maximum deviation of the object, reduction power of the lens501, and pixel pitch in the photodetector arrays 11.

When all the values Call (m, n) are obtained for the entire search range(Yes in S611), the smallest value Call (m, n) is selected among all thevalues Call (m, n) which are now stored in the calculating portion 15B.Then, the value (m,n) for the selected value Call (m, n) is determined.The thus determined position (m, n) indicates the matching position. Atthis point, the RGB template matching process of S600 ends.

Next, as shown in FIG. 21, the image shift amount(m²=n²)^({fraction (1/ 2)}) is calculated in S506 based on the values m,n which have been determined in S600. The image shift amount indicatesthe amount, by which the object image in the block deviates from theideal position. It is noted that in the RGB template matching process ofS600, the processes of S610-S612 are executed by the program controlportion 15A controlling the processing elements 400 to perform parallelprocessing. The calculation processes of S613 and S506 are executed bythe calculating portion 15B. It is noted that by performing the RGBtemplate matching processes of S600 for all the blocks in the lightreceiving surfaces 11 r, 11 g, and 11 b, it is possible to obtain thepositional shift amount of the objects appearing in the respectiveblocks.

The shift registers 410 r, 410 g, and 410 b transfer RGB data for oneframe to the processing elements 400. Then, while the processingelements 400 and the calculating portion 15B perform their calculationsin S600-S506 onto the received data, the shift registers 410 r, 410 g,and 410 b move on to the process of the next frame in S505, and executeprocesses in S501-S503. After completing a calculation process inS600-S506, the processing elements 400 and the calculating portion 15Bmove on to the process of the next frame in S507, and receive RGB imagedata for the next frame from the shift registers 410 r, 410 g, and 410 bin S504. By repeating the above-described processes, the processingelements 400 and the calculating portion 15B can perform calculations onthe present frame in S600-S506 while the shift registers 410 r, 410 g,and 410 b transfer RGB image data for the next frame in S501-S503,thereby reducing wasteful wait time.

Next, another embodiment (fourth embodiment) that employs a plurality ofphotodetector arrays will be described with reference to FIGS. 23-26.

FIG. 23 is an explanatory diagram showing the overall construction ofthe system 10 for detecting a high-speed moving object, and FIG. 24 is adetailed block diagram of the system.

An image of an object 2000 c passes through a lens 501 and is split intotwo directions by a beam splitter 504. The split images are guidedthrough shutters 505A and 505B to photodetector arrays 11A and 11B. Bycontrolling the shutters 505A and 505B with the control circuit 15, theimages of the object 2000 c at different times (different frames) areobtained by the photodetector arrays 11A and 11B. More specifically, thephotodetector array 11A obtains an image P(t) at a time t (frame t),while the photodetector array 11B obtains an image P(t+Δt) for anothertime t+Δt (frame t+Δt). By controlling the shutters 505A and 505B, it ispossible to set the length of the Δt (frame rate) to an extremely shortperiod of time.

As in the second embodiment, as shown in FIG. 24, the photodetectorarrays 11A and 11B have the same construction as the photodetector array11 in the first embodiment. Analog-to-digital converter arrays 13A and13B have the same construction as the analog-to-digital converter array13 in the first embodiment. As in the second embodiment, the parallelprocessing system 14 includes N1 columns×N2 rows of processing elements400 which are arranged two-dimensionally to correspond one-on-one to theN1×N2 photodetectors 120 in each of the photodetector arrays 11A and11B. The high-speed vision sensor 10 of the present embodiment isfurther provided with N2 shift register lines 420A in one to onecorrespondence with the N2 rows of processing elements. The high-speedvision sensor 10 of the present embodiment is further provided withother N2 shift register lines 420B in one to one correspondence with theN2 rows of processing elements. The shift register lines 420A and 420Bare connected to the analog-to-digital converter arrays 13A and 13B,respectively. As in the second embodiment, N1 shift registers 410A, eachhaving a plurality of bits (four bits in this example), are connected inseries in each shift register line 420A. N1 shift registers 410B, eachhaving a plurality of bits (four bits in this example), are connected inseries in each shift register line 420B. Each processing element 400 isconnected to both of a corresponding shift register 410A and acorresponding shift register 410B.

With this configuration, two images obtained by the photodetector arrays11A and 11B are converted to digital signals by the correspondinganalog-to-digital converter arrays 13A and 13B. The digital signals arefed to the single parallel processing system 14. In the parallelprocessing system 14, the digital signals are transferred by thecorresponding shift registers 410A and 410B to the processing elements400, and are subjected to calculation operations. In this way, thehigh-speed vision sensor 10 of the present embodiment can detectmovement of a high-speed moving object that has been difficult to berecognized based on images picked up at a conventional frame rate.Additionally, image processing can be performed without delays at aframe rate higher than the conventional frame rate.

Next, the operations of the present embodiment will be described withreference to the flowcharts in FIGS. 25-26.

It is noted that according to the present embodiment, an image for acertain frame at a time t is transferred. Then, while an image for thenext frame (t+Δt) is being transferred, calculation is performed on twoimages for time t and the previous time (t−Δt) to determine the velocityof the moving object at time t. While an image for the next frame (t+2Δt) is being transferred, calculation is performed on two images fortime (t+Δt) and time t to determine the velocity of the moving object attime (t+Δt).

More specifically, first, in S701, image data t(x, y) for frame t attime t is obtained from the corresponding camera 11 a. Next, in S702 andS703, the image data of the frame at time t is transferred using thecorresponding shift registers 410A up to the processing element 400 atthe corresponding position (x, y). Next, in S704, at the transferredposition (x, y), image data t(x, y) is transferred from thecorresponding shift register 410A to the corresponding processingelement 400. It is noted that image data t′(x, y) for the frame at time(t−Δt), which has been obtained by the photodetector array 11B at time(t−Δt), has already been transferred by the shift registers 410B andstored in the processing element 400.

Next, a matching process for images over A t is performed in S800.

During the Δt matching process, as shown in FIG. 26, in order to findwhat portion of the image in (t−Δt) has a high match with a single block(3×3 pixel block, for example) in the image at time t, a correlationvalue C(m, n), where m and n are first initialized to zero (0), iscalculated with respect to the block in the image of time t and acorresponding current block in the image of time (t−Δt).

It is noted that the value C(m, n) is expressed by the followingequation:

C(m,n)=Σ|t′(x,y) −t(x+m,y+n)|

wherein x, y represents all the pixels located in the current block. Thecorrelation value C(m, n) can be calculated in the same manner thecorrelation value C(j) is calculated in the second embodiment. Thecalculated correlation value C(m, n) is transferred to and stored in thecalculating portion 15B.

Next, the values m and n are changed within the entire search range (Noin S811 and S812). More specifically, image data for time t istransferred up, down, left, or right between the register matrices 401of neighboring processing elements 400.

Next, the correlation value C(m, n) is calculated again in S810. Thus,calculation of the correlation value C(m, n) is repeated within theentire search range while transferring image data for the time t betweenneighboring processing elements 400 (no in S811, S812). It is noted thatthe search range is determined dependently on conditions such as anestimated maximum velocity of the object, reduction power of the lens501, and pixel pitch of the photodetector arrays 11A and 11B. Becausethe high-speed vision sensor 10 of the present embodiment can achieve afast frame rate by setting the shutter time interval Δt to a very smallvalue, it is possible to set both m and n in a small range, such as −1,0, and +1.

After all the correlation values C(m, n) are obtained for the entiresearch range (Yes in S811), the smallest value C(m,n) is selected inS813 among all the correlation values C(m, n) (where m=−1, 0, +1 andn=−1, 0, +1) which are currently stored in the calculating portion 15B.Then, the value (m, n) for the smallest value C(m, n) is determined. Thethus determined value (m, n) indicates the matching position of theblock in the image at time t with respect to the image at time (t−Δt)

Thus, the matching process of S800 ends.

Next, as shown in FIG. 25, in S706, the shift amount (m²+n²)½ iscalculated based on the value (m, n) determined as described above. Thisshift amount indicates the deviation of the object image in the block.The velocity of the object is calculated based on this deviation. It isnoted that during the Δt image matching process of S800, processesS810-S812 is executed by the control portion 15A controlling theprocessing elements 400 to perform parallel processing. The calculationsin S813 and S706 are executed by the calculating portion 15B. Further,by performing the matching process of S800 for all the blocks in thelight receiving surfaces, it is possible to determine in S706 thepositional shift amount, during the time At, of the objects appearing inthe respective block images and the velocities of the objects.

As described above, while calculation process is being performed inS800-S706 onto the images at time t and time (t−Δt), the light receivingportion 11B and the corresponding shift registers 410B move on to theprocess of the next frame (t+Δt) in S705, thereby transferring the imagefor the time (t+Δt) to the parallel processing system 14 by executingthe processes of S701-S703. After completing the calculation of imagesat time t and time (t−Δt) in S800-S706, the processing elements 400 andthe calculating portion 15B move on to process the next frame in S707.When image data for the next frame time (t+Δt) is transferred to theregister matrices 401 from the shift registers 410B in S704, theprocessing elements 400 and the calculating portion 15B performcalculation for time t and time (t+Δt) in S800-S706. In this way, inputof frame images is repeatedly alternated between the photodetectorarrays 11A and 11B. At the same time, calculations are repeatedlyperformed on two consecutive frames. Hence, the processing elements 400and calculating portion 15B can perform calculations of S800-S706 whilethe shift registers 410A or 410B transfer image data for the next frame(next time) in S701-S703, thereby reducing wasteful wait time.

The high-speed vision sensor 10 of the present embodiment can be appliedto an automatic tracking system, for example, and can be used to controla camera-mounting unit (external device 1000), which mounts thereon thephotodetector arrays 11A and 11B, with respect to the matched imageposition.

It is noted that in the example described above, the present embodimentis provided with the two photodetector arrays. However, the embodimentcan include three or more photodetector arrays. In this case, theparallel processing system 14 has to be provided with a group of shiftregisters dedicated for each photodetector array.

In the high-speed vision sensor 10 of the second through fourthembodiments described above, when a plurality of photodetector arrays 11a are used, a group of shift registers 410 is provided as beingdedicated for each photodetector array 11. For this reason, thehigh-speed vision sensor 10 can perform transfer operation andprocessing operation, in parallel, onto the images obtained at therespective photodetector arrays, thereby achieving efficient transfer.Accordingly, the high-speed vision sensor 10 is particularly suitablefor such processes as three-dimensional vision, color image processing,and detection of high-speed moving object.

Next, a fifth embodiment of the present invention will be described withreference to FIGS. 27-32.

Various types of image processes require data, separate from input imagedata, such as image data other than the input image data or parametersto be used for image processes. The present embodiment is provided forenabling faster processing of those various image process calculationsthat include the step of transferring the separate data.

FIG. 27 shows the construction of the high-speed vision sensor 10according to the present embodiment. The high-speed vision sensor 10 ofthe present embodiment is different from that in the first embodiment(FIG. 1) in that the high-speed vision sensor 10 of the presentembodiment is provided with a data buffer/data generator 17, and thatthe parallel processing system 14 has shift registers 411 fortransferring data from the data buffer/data generator 17 to theprocessing elements 400.

More specifically, the parallel processing system 14 of the presentembodiment is provided with N2 shift register lines 430, in addition tothe N2 shift register lines 420. The N2 shift register lines 430 arearranged in one to one correspondence with the N2 rows of processingelements. The N2 shift register lines 430 are connected to the databuffer/data generator 17. As in the shift register lines 420, each shiftregister line 430 includes N1 shift registers 411 which are connected inseries. Each shift register 411 has plural bits (four bits in thepresent embodiment). Each processing element 400 is connected to both ofthe corresponding shift register 410 and the corresponding shiftregister 411.

The data buffer/data generator 17 is for transmitting and receivingpredetermined external data (process data and/or control data), whichare required for image processing calculations, to and from the shiftregisters 411 according to instructions issued by the control circuit15.

It is noted that the process data is required for image processing.Examples of the process data include coefficient data. The control dataindicates a control description of the calculation. Examples of thecontrol data include data indicative of whether to or not to calculate.The shift registers 411 transfer the external data, received from thedata buffer/data generator 17, to the processing elements 400.

As shown in FIG. 28, the data buffer 17 can be formed on the samecircuit board as the parallel processing system 14. Or, the data buffer17 can be formed on the same circuit board as the control circuit 15.The data buffer 17 can be integrated on either the parallel processingsystem 14 or the control circuit 15.

According to the present embodiment, as shown in FIG. 29, eachprocessing element 400 is connected to both of the shift register 411and the shift register 410. Hence, in response to instructions from thecontrol circuit 15, the data buffer/data generator 17 prepares controldata and/or processing data according to the contents of the process tobe performed. The data buffer/data generator 17 then transfers thecontrol data and/or processing data via the shift registers 411 to theregister matrices 401 of the processing elements 400. This data transferoperation is performed in parallel to image data transfer operation. Theshift registers 411 operate substantially in the same manner as theshift registers 410, and therefore description of the operations of theshift registers 411 is omitted.

As in the first through fourth embodiments, the high-speed vision sensor10 of the present embodiment can perform the method called SIMD forcontrolling all of the processing elements 400 to perform the samecalculations. Additionally, the high-speed vision sensor 10 of thepresent embodiment can allow the respective processing elements 400 toperform different operations by transferring non-uniform control data tothe processing elements 400 using the shift registers 411. Accordingly,the high-speed vision sensor 10 of the present embodiment can performmore flexible processing.

Next will be described a detailed description of one example of theprocess that uses the shift registers 411. It is noted that the humaneye has a function for observing the center portion of an image in highresolution and the peripheral portion in low resolution. The exampledescribed below relates to an image processing that attains thisfunction.

This process can be implemented by differing the processing descriptionfor the central portion of an image from that for the peripheralportions of the same image. This process can be implemented also bydiffering the weight variables to be used for the central portion of animage from those to be used for the peripheral portions of the sameimage.

For example, in order to lower the resolution in the peripheral area, itis conceivable to perform a process called smoothing by performingcalculations between four neighboring pixels as indicated below:${I^{\prime}\left( {x,y} \right)} = \frac{\begin{Bmatrix}{{{D1} \times I\left( {x,y} \right)} + {{D2} \times {I\left( {x,{y + 1}} \right)}} + {{D3} \times I\left( {x,{y - 1}} \right)} +} \\{{{D4} \times I\left( {{x + 1},y} \right)} + {{D5} \times {I\left( {{x - 1},y} \right)}}}\end{Bmatrix}}{{D1} + {D2} + {D3} + {D4} + {D5}}$

wherein I(x, y) is input image data received at the position (x, y), andI′(x, y) is resultant image data obtained at the position (x, y) by thepresent processing. D1-D6 are weight variables used for obtaining aweighted average. It is possible to adjust the degree of smoothing bychanging the amounts of the weight variables for each pixel. Forexample, D1 is set to one (1) and D2 through D5 are set to zero (0) inthe center portion where no smoothing is desired to be performed, whileD1 is set to two (2) and D2 through D5 are set to one (1) in theperipheral areas in order to provide much smoothing in those areas. Inorder to attain a proper smoothing operation, the amounts of the weightvariables are previously determined for the respective pixels accordingto the positions of the pixels. The weight variables are then stored inthe data buffer/data generator 17. At the time of calculations, data ofthose variables are transferred to the processing elements 400 via theshift registers 411, enabling the respective processing elements toperform different calculations from one another.

In this way, it is possible to perform calculations using differentfiltering functions for the respective pixels. Hence, when it is moreimportant to attain high-speed calculations than to attain highresolution, for example, it is possible to first calculate the addingaverage of image data for every 2×2=4 pixels and then to perform, at therespective four pixels, detection of a vertical edge, a horizontal edge,a right diagonal edge, and a left diagonal edge, thereby detecting edgesin four directions simultaneously.

Next, the operations of the high-speed vision sensor 10 will bedescribed with reference to the flowchart in FIG. 30.

First, in S901, image data from the photodetector array 11 is outputtedto the parallel processing system 14 via the analog-to-digital converterarray 13. Simultaneously, the data buffer/data generator 17 iscontrolled to output process commands to the parallel processing system14.

The process command indicates a 3×3 weighting mask. To performhigh-resolution filtering on the center portion and low-resolutionfiltering on the peripheral portion, for example, the weighting mask tobe transferred to each pixel in the center portion is merely formultiplying the image value of the subject pixel by eight (8). Theweighting mask to be transferred to each pixel in the peripheral portionis for performing a smoothing operation with eight neighboring pixelsthat are located to the up, down, left, and right of the subject pixel.More specifically, the data buffer/data generator 17 prepares two typesof mask data: mask A=(0, 0, 0, 0, 8, 0, 0, 0, 0) and mask B=(0, 1, 0, 1,4, 1, 0, 1 0). Mask A is for multiplying the image value of the subjectpixel by eight (8), while mask B is for smoothing the subject pixel withthe neighboring pixels that are located to the up, down, left, and rightof the subject pixel. It is now assumed that the parallel processingsystem 14 has 128×128 processing elements 400. That is, the coordinates(x, y) for all the processing elements are in the range of ×=0˜127 andy=0˜127. In this case, the mask A is used for the processing elements400 that are located in the center portion of the parallel processingsystem 14. More specifically, the mask A is used for those processingelements 400 that have coordinates (x, y) in the range of ×=32˜95 andy=32˜95. On the other hand, mask B is used for the peripheral portion ofthe parallel processing system 14. More specifically, the mask B is usedfor those processing elements 400 that are located at coordinates (x, y)in the range of ×=0˜31 or 96˜127 and y=0˜31 and 96˜127.

In S902-S903, image data I(x, y) from the analog-to-digital converterarray 13 is transferred through the shift registers 410 to theprocessing elements 400 at corresponding positions (x, y). At the sametime, masks A and B are transferred from the buffer/data generator 17through the shift registers 411 to the processing elements 400 at thecorresponding positions (x, y). It is noted that the mask A istransferred to coordinate positions (x, y) where 32≦×≦95 and 32≦y≦95.The mask B is transferred to coordinate positions (x, y) where 0≦×≦31 or96≦×≦127 and y=0≦y≦31 or 96≦y≦127.

When the transfer is completed, image data I(x, y) is transferred fromeach shift register 410 to the register matrix 401 of the correspondingprocessing element 400 in S904. Next, in S905, the mask A or B istransferred from each shift register 411 to the register matrix 401 ofthe corresponding processing element 400.

Next, in S907, each processing element 400 receives input image dataI(x−1, y−1), I(x, y−1), I(x+1, y−1), I(x−1, y), I(x+1, y), I(x−1, y+1),I(x, y+1), and I(x+1, y+1) from its eight neighboring pixels, and usesthe mask A or B, which is stored in S905, to perform the followingcalculations onto the eight neighboring input image data and its owninput image data I(x, y):

I′(x,y)=mask(0)×I(x−1,y−1)

+mask(1)×I(x,y−1)

+mask(2)×I(x+1,y−1)

+mask(3)×I(x−1,y)

+mask(4)×I(x,y)

+mask(5)×I(x+1,y)

+mask(6)×I(x−1,y+1)

+mask(7)×I(x,y+1)

+mask(8)×I(x+1,y+1)

 I″(x,y)=I′(x,y)/8

It is noted that the processing elements 400 located in the centerportion receive the mask A=(0, 0, 0, 0, 8, 0, 0, 0, 0). Accordingly,those processing elements 400 perform the above-described calculation bysetting “mask (0)” through “mask (3)” and “mask (5)” through “mask (8)”to zero (0) and by setting “mask (4)” to eight (8). The processingelements 400 located in the peripheral portions receive the mask B=(0,1, 0, 1, 4, 1, 0, 1, 0). Accordingly, those processing elements 400perform the above-described calculation by setting mask (0), mask (2)mask (6), and mask (8) to zero (0), by setting mask (1) mask (3), mask(5), and mask (7) to one (1); and by setting mask (4) to four (4). Inthis way, the present embodiment can perform image processingcalculations by setting different mask coefficients to respectivepixels.

It is noted that among the image data for the eight neighboring pixels,four sets of image data I(x, y−1), I(x−1, y), I(x+1, y), and I(x, y+1)are transferred to each processing element 400 directly from the shiftregister matrices of the neighboring processing elements 400 on the foursides. The remaining four sets of image data I(x−1, y−1), I(x+1, y−1),I(x+1, y+1), and I(x+1, y+1) are transferred to each processing element400 by repeating two times the transfer operation between neighboringprocessing elements.

Each processing element 400 outputs the resultant image data I″(x, y−1)to the calculating portion 15 b. The image data is subjected to anecessary image process in the calculating portion 15B. It is noted thatthe resultant image data I″(x, y) indicates an image whose centerportion is not changed from the original state but whose peripheralportion is smoothed. By using this image, it is possible to performmatching operation while allowing the peripheral portion of the image tobe unfocused, similarly to the process achieved by a human eye. Thecalculating portion 15B can therefore know the general position and thegeneral shape of an object that is located in the peripheral portion ofthe image by making a general matching of the object based on theresultant image I″(x, y). Based on the result of the general matchingoperation, control operations are performed to move an external device1000, which is the camera-mounting device mounting the photodetectorarray 11, in order to position the object in the center of the camera,enabling more detailed matching thereafter.

The shift registers 410 and 411 transfer image data and mask data to theprocessing elements. While the processing elements 400 performcalculations on the received image and mask data in S907, the shiftregisters 410 and 411 move on to the process of the next frame in S906and execute processes in S901-S903. After completing the calculatingprocess in S907, the processing elements 400 move on to the next frameprocess in S908. Image data for the next frame is transferred from theshift registers 410 to the register matrices 401 of the processingelements, and mask data is transferred from the shift registers 411 tothe register matrices 401 of the processing elements (S904 and S905). Byrepeating the above-described processes, the processing elements 400perform calculations in S907 while the shift registers 410 and 411transfer data for the next frame in S901-S903, thereby reducing theamount of unnecessary wait time.

FIG. 31 is a timing chart showing timings of the transfer processes andthe calculating processes executed by a comparative example of aparallel processing system that is not provided with shift registers 410or 411. FIG. 32 is a timing chart showing timings of the transferprocesses and the calculating processes executed by the parallelprocessing system of the present embodiment. Because the parallelprocessing system of the comparative example is provided with no shiftregisters 410 or 411, transfer operations are performed by theprocessing elements. Accordingly, as shown in FIG. 31, this system hasto perform transfer of an image, transfer of a process/control signal,and image calculations in a time-sharing manner.

Accordingly, the time (one frame) required from the beginning of theprocess of one image to the beginning of the process of the next imageis at least the <image transfer time>+<process/control signal transfertime>+<image processing time>.

Contrarily, the parallel processing system of the present embodiment canperform the image transfer, the process/control signal transfer, and theimage calculation independently and in parallel with one another, asshown in FIG. 32. Accordingly, one frame can be reduced to the longestone among the <image transfer time>, <process/control signal transfertime>, and <image processing time>. Hence, real-time processing can beachieved.

As described above, according to the present embodiment, image data andprocessing/control data are transferred by the shift registers 410 and411, respectively.

Accordingly, it is possible to perform transfer operations and thecalculation operations in parallel. It is possible to perform, at ahigh-rate of speed, those processing operations that require transfer ofdata from an external source.

Next, a sixth embodiment will be described with reference to FIGS.33-35.

FIGS. 33 is a block diagram showing the high-speed vision sensor 10 ofthe present embodiment. FIG. 34 shows the construction of the processingelement 400 and a shift register 412. The high-speed vision sensor 10 ofthe present embodiment is different from that of the fifth embodimentshown in FIGS. 27 and 29 in that one shift register line 440 fortransferring both of image data and commands is provided for each row jof processing elements in the parallel processing system 14. In otherwords, N2 shift register lines 440 are provided in one to onecorrespondence with the N2 rows of processing elements. In addition, thehigh-speed vision sensor 10 is provided with an image buffer array 18and a mixer array 19. The image buffer array 18 includes N2 imagebuffers 601, each for temporarily storing image data sent from thecorresponding analog-to-digital converter 210. The mixer array 19includes N2 time-sharing mixers 602, each for combining signals from thecorresponding image buffer 601 and signals from the data buffer/datagenerator 17 on a time-sharing basis. The N2 shift register lines 440are connected one-on-one to the N2 time-sharing mixers 602 in the mixerarray 19. N1 shift registers 412 are connected in series in each of theshift register lines 440. Each shift register 412 has a plurality ofbits (four bits in this example). Each processing element 400 isconnected to a corresponding shift register 412.

Unlike the high-speed vision sensor 10 in the fifth embodiment, whereinimage data and process/control data are transferred separately by theshift registers 410 and 411, the high-speed vision sensor 10 of thepresent embodiment mixes, in time-sharing basis, the image data and theprocess/control data by the time-sharing mixers 602, and transfers themixed data with the shift registers 412 only. It is noted that the speedof the overall system is determined by the conversion rate of theanalog-to-digital converters 210 (1 Mbps (bits-per-second)). Themultiplex transfers during one frame are sufficiently feasible becausethe transfer rate of the shift registers 412 is about 10-100 Mbps.

FIG. 35 is a timing chart showing timings of transfer processes andcalculating processes by the parallel processing system of the presentembodiment. As with the fifth embodiment whose processing timing isshown in FIG. 32, the parallel processing system of the presentembodiment can perform a pipeline process by performing the imagecalculations and transfers in parallel. Unlike the processing timing ofthe comparative example shown in FIG. 31, the high-speed vision sensor10 of the present embodiment can shorten the processing time to thelonger one of (the time required for transferring image data andprocess/control signals) and (the time required for processing imagedata). Hence, the high-speed vision sensor 10 of the present embodimentcan achieve real-time processing.

As described above, according to the fifth and sixth embodiments, notonly image data but also process/control data, stored in the data buffer17, is transmitted to the processing elements 400 via the shiftregisters, thereby achieving high-speed and flexible calculations. Theprocess/control data can be transferred using either the dedicated shiftregisters, as described in the fifth embodiment, or the same shiftregisters used for transferring the image data, as described in thesixth embodiment.

The high-speed vision sensor of the present invention is not limited tothe above-described embodiments, but can be modified in various manners.

For example, in the second through fourth embodiments, image data istransferred from the plural photodetector arrays 11 using the pluralgroups of shift registers 410. However, it is possible to transfer, in atime-sharing basis, image data from the plural photodetector arraysusing only the single group of shift registers 412 by using the imagebuffer array 18 and the mixer array 19 as described in the sixthembodiment.

In the first through sixth embodiments, the analog-to-digital converter210 includes the charge amp 221 _(j). However, the analog-to-digitalconverter 210 and the charge amp 221 can be provided separately, asshown in FIG. 36. As shown in this figure, an amp array 12 is connectedto the photodetector array 11. The amp array 12 has N2 charge amps 221.The analog-to-digital converter array 13 is provided between the amparray 12 and the parallel processing system 14. The analog-to-digitalconverter array 13 has N2 analog-to-digital converters 210. With thisconstruction, each amp 220 in the amp array 12 successively convertscharges, outputted from the N1 photodetectors 120 on the correspondingrow 110 of the photodetector array 11, into voltage signals. Theseanalog voltage signals are outputted to the correspondinganalog-to-digital converter 210 in the analog-to-digital converter array13. The analog-to-digital converter 210 successively converts the analogvoltage signals to digital signals, and supplies the digital signals tothe parallel processing system 14.

The configurations in the fifth and sixth embodiments can also beapplied to a high-speed vision sensor 10 that has a plurality ofphotodetector arrays 11 as described in the second through fourthembodiments.

INDUSTRIAL APPLICABILITY

The high-speed vision sensor of the present invention can be employed ina wide variety of vision recognition processes, including FA robotcontrol, three-dimensional vision, color image processing, andrecognition of a high-speed moving object.

What is claimed is:
 1. A high-speed vision sensor, comprising; at leastone photodetector array, each having a plurality of photodetectors whichare arranged two-dimensionally in a plurality of rows and in a pluralityof columns; an analog-to-digital converter array having a plurality ofanalog-to-digital converters which are arranged one-dimensionally suchthat each analog-to-digital converter corresponds to one row in the atleast one photodetector array, each analog-to-digital converterconverting, into digital signals, analog signals which are successivelyoutputted from the photodetectors in the corresponding row; a parallelprocessing system, including a parallel processing element array and ashift register array, the parallel processing element array having aplurality of processing elements which are arranged two-dimensionally ina plurality of rows and in a plurality of columns and in one-to-onecorrespondence with the plurality of photodetectors in the at least onephotodetector array, each processing element performing a predeterminedcalculation on digital signals transferred from the analog-to-digitalconverter array, the shift register array having a plurality of shiftregisters which are disposed in one-to-one correspondence with theplurality of analog-to-digital converters and in one-to-onecorrespondence with the plurality of rows of processing elements, eachshift register successively transferring digital signals, which arereceived from the corresponding analog-to-digital converter and whichare equivalent to signals outputted from the photodetectors in acorresponding photodetector row, to predetermined processing elements inthe corresponding row; and a control circuit controlling thephotodetector array and the analog-to-digital converter array to outputdigital signals for a single frame and controlling the shift registerarray to transfer the digital signals of the single frame to theparallel processing element array, and thereafter controlling thephotodetector array and the analog-to-digital converter array to outputdigital signals for the next frame and controlling the shift registerarray to transfer the digital signals of the next frame to the parallelprocessing element array, while simultaneously controlling the parallelprocessing element array to perform the predetermined calculation ontothe single frame.
 2. A high-speed vision sensor as claimed in claim 1,wherein the at least one photodetector array includes a plurality ofphotodetector arrays, and wherein the parallel processing systemincludes, in correspondence with each processing element row, aplurality of lines of shift registers, the number of the plurality oflines being equal to the number of the plurality of photodetectorarrays.
 3. A high-speed vision sensor as claimed in claim 2, wherein theplurality of photodetector arrays are disposed at positions differentfrom one another, and wherein the control circuit includes: a parallelprocessing control portion controlling the respective lines of shiftregisters to transfer images which are taken at different positions andoutputted from the corresponding photodetector arrays, and controllingthe parallel processing system to perform calculations, based on imagesignals obtained by the plurality of photodetector arrays, to determinethe amount of positional shift, of an object, between its images takenby the plurality of photodetector arrays; and a calculating portioncalculating three-dimensional positional information of the object basedon the determined amount of positional shift and information on theposition of each photodetector array and on the direction in which eachphotodetector array takes images.
 4. A high-speed vision sensor asclaimed in claim 3, wherein the plurality of photodetector arrays arearranged such that data transfer directions, along which the rows of thephotodetectors in the respective photodetector arrays extend to transferdata from the photodetectors, are lined up in the same direction andsuch that corresponding photodetectors in the plural photodetectorarrays are positioned to be shifted from one another in the datatransfer direction, and wherein the parallel processing control portioncontrols the parallel processing system to calculate, based on the imagesignals obtained by the plural photodetector arrays, the amount ofpositional shift along the data transfer direction among the imagestaken by the plural photodetector arrays.
 5. A high-speed vision sensoras claimed in claim 4, further comprising a pixel shift amount storagedevice storing the amount of positional shift, along the data transferdirection, which is calculated by the parallel processing system withrespect to the plurality of images taken by the plurality ofphotodetector arrays, wherein the parallel processing control portioncontrols, based on the stored positional shift amount, the transferposition of the processing elements, to which each line of shiftregisters transfers an image signal.
 6. A high-speed vision sensor asclaimed in claim 2, further comprising: a timing controller thatcontrols the plurality of photodetector arrays to take images at timingsindependent from one another; and a beam splitter that enables theplurality of photodetector arrays to pick up images from the samedirection, and wherein the control circuit includes a parallelprocessing control portion controlling the plural lines of shiftregisters to transfer images taken by the respective photodetectorarrays at the different times, and controlling the processing elementsto perform calculation onto the images taken at the different times. 7.A high-speed vision sensor as claimed in claim 2, further comprising afiltering/beam splitter mechanism enabling the plurality ofphotodetector arrays to pick up, from the same direction,color-separated images, of an object, which have colors different fromone another, and wherein the control circuit includes a parallelprocessing control portion controlling the plural lines of shiftregisters to transfer the color-separated images outputted from therespective photodetector arrays and controlling the processing elementsto perform calculations onto the color-separated images.
 8. A high-speedvision sensor as claimed in claim 1, further comprising data supplymechanism supplying predetermined data required for image processingcalculation, wherein each of the plurality of shift registerssuccessively transfers output signals, outputted from the correspondinganalog-to-digital converter, and the predetermined data, supplied fromthe data supply mechanism, to the predetermined processing elements inthe corresponding processing element row, and wherein the controlcircuit controls the photodetector array, the analog-to-digitalconverter array, the parallel processing system, and the data supplymechanism.
 9. A high-speed vision sensor as claimed in claim 8, whereinthe data supply mechanism includes a time-sharing mixing unit combining,according to a time-sharing method, output signals outputted from theanalog-to-digital converters and the predetermined data.
 10. Ahigh-speed vision sensor as claimed in claim 1, wherein the parallelprocessing system further includes a plurality of data-transfer shiftregisters which are arranged in one-to-one correspondence with theplurality of rows of processing elements, each data-transfer shiftregister supplying predetermined data to the respective processingelements in the corresponding row.
 11. A high-speed vision sensor asclaimed in claim 1, wherein the at least one photodetector arrayincludes a plurality of photodetector arrays, the parallel processingsystem including a single line of shift register for each processingelement row, the single line of shift register being used in atime-sharing manner to transfer the output signals from the plurality ofphotodetector arrays.